Carriage state detection method and device

Abstract

The present invention provides a method for detecting the state of a carriage, comprising: obtaining a current frame image of the top of the carriage; using a convolutional or fully connected network to obtain the features of the current frame image; determining the closed state of the carriage according to the proportion of the covering in the current frame image to the current frame image; determining the cargo state of the carriage according to the proportion of the carriage in the current frame image to the current frame image; and sending corresponding alarm information according to the closed state and the cargo state. The present invention can obtain the specific closed degree and cargo degree of the carriage, and can provide users with a more detailed carriage state, which is conducive to reducing the scattered cargo.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a method and device for detecting a carriage state. Background Art

[0002] During the transportation process, road transport vehicles usually need to close the compartment to prevent gravel, soil or domestic garbage in the compartment from being scattered on the road, damaging the road environment and blocking road traffic. Therefore, it is necessary to detect the closed state of road transport vehicles.

[0003] In the process of realizing the present invention, the inventors found that there are at least the following technical problems in the prior art: the detection of the closed state of a road transport vehicle in the prior art can usually only detect the closed or unclosed state and the loaded or unloaded state, but cannot specifically determine the degree of closedness and the degree of loading. Summary of the invention

[0004] The carriage state detection method and device provided by the present invention can obtain the specific degree of closure and cargo loading of the carriage, and can provide users with a more detailed carriage state, which is conducive to reducing the scattering of cargo.

[0005] In a first aspect, the present invention provides a method for detecting a carriage state, comprising:

[0006] Get the current frame image of the top of the carriage;

[0007] Using a convolutional or fully connected network to obtain features of the current frame image;

[0008] Determining the closed state of the compartment according to the proportion of the covering in the current frame image to the current frame image;

[0009] Determining the cargo state of the carriage according to the proportion of the carriage in the current frame image;

[0010] Corresponding alarm information is sent according to the closed state and the cargo state.

[0011] Optionally, using a convolutional or fully connected network to obtain features of the current frame image includes:

[0012] The YOLO detection network is used to obtain the carriage position in the current frame image, and the carriage position is segmented to obtain the area to be processed;

[0013] The area to be processed is processed using a convolutional or fully connected network to obtain features of the current frame image.

[0014] Optionally, using a convolutional or fully connected network to obtain features of the current frame image includes:

[0015] Encoding the current frame image to form an image to be processed;

[0016] Performing convolution or full connection processing on the image to be processed to obtain a target category of the image to be processed, a confidence level corresponding to the target category, and a target position corresponding to the target category;

[0017] Determining a feature map to be processed according to the target category, confidence level, target position, and image to be processed;

[0018] Performing convolution or full connection processing on the feature map to be processed to obtain a probability map corresponding to the target category and a classification feature map corresponding to the target category;

[0019] Determine a target feature map based on the probability map and the classification feature map;

[0020] The target feature map is processed using a convolutional or fully connected network to obtain the features of the current frame image.

[0021] Optionally, determining the loading state and / or closing state of the carriage includes:

[0022] A neural network model is used to output confidence values ​​of a plurality of degree intervals, wherein the degree intervals represent the degree of the cargo loading state and / or the closed state of the compartment;

[0023] The multiple values ​​are weighted and summed using the degree interval as the weight value to obtain the cargo loading state and / or the closed state of the vehicle compartment.

[0024] Optionally, it also includes:

[0025] Acquire the loading state and / or closing state corresponding to the multiple frames of images;

[0026] Obtaining the number of images having the same loading state and / or closure state;

[0027] When the number of the images is greater than the predetermined threshold, the loading state and / or the closing state of the compartment is determined according to the loading state and / or the closing state corresponding to the number of the images.

[0028] In a second aspect, the present invention provides a vehicle compartment state detection device, comprising:

[0029] The vehicle-mounted camera is set at the rear of the vehicle head and is used to obtain the current frame image of the top of the vehicle compartment;

[0030] An algorithm operation module is communicatively connected to the vehicle-mounted camera, wherein the algorithm operation module includes: a feature acquisition submodule, used to obtain the features of the current frame image using a convolutional or fully connected network; a state acquisition submodule, used to determine the closed state of the compartment according to the proportion of the covering in the current frame image occupied by the current frame image; and determine the cargo state of the compartment according to the proportion of the compartment in the current frame image occupied by the current frame image.

[0031] The alarm module is used to send corresponding alarm information according to the closed state and the cargo state.

[0032] Optionally, the feature acquisition submodule includes:

[0033] A segmentation unit, used for acquiring the carriage position in the current frame image by using a YOLO detection network, and segmenting the carriage position to obtain a to-be-processed area;

[0034] The feature acquisition unit is used to process the area to be processed by using a convolutional or fully connected network to obtain the features of the current frame image.

[0035] Optionally, the feature acquisition submodule includes:

[0036] An encoding unit, used for encoding the current frame image to form an image to be processed;

[0037] an identification unit, configured to perform convolution or full connection processing on the image to be processed to obtain a target category of the image to be processed, a confidence level corresponding to the target category, and a target position corresponding to the target category;

[0038] A pooling unit, used to determine a feature map to be processed according to the target category, confidence, target position and image to be processed;

[0039] A classification unit, used for performing convolution or full connection processing on the feature map to be processed to obtain a probability map corresponding to the target category and a classification feature map corresponding to the target category;

[0040] A target unit, used to determine a target feature map based on the probability map and the classification feature map;

[0041] A feature acquisition unit is used to process the target feature map using a convolutional or fully connected network to acquire the features of the current frame image.

[0042] Optionally, the status acquisition submodule includes:

[0043] A multi-output unit, used to output confidence values ​​of multiple degree intervals using a neural network model, wherein the degree intervals represent the degree of the cargo loading state and / or the closed state of the compartment;

[0044] The averaging unit is used to perform weighted summation on the multiple values ​​using the degree interval as the weight value to obtain the cargo loading state and / or the closed state of the carriage.

[0045] Optionally, it also includes:

[0046] A data collection module, used to obtain a loading state and / or a closing state corresponding to a plurality of frames of images;

[0047] A data statistics module, used for obtaining the number of images with the same loading state and / or closed state;

[0048] A state determination module is used to determine the loading state and / or closing state of the compartment according to the loading state and / or closing state corresponding to the number of images when the number of images is greater than the predetermined threshold.

[0049] In the technical solution provided by the present invention, by making a more precise judgment on the closed state and the cargo state of the vehicle, a warning message can be issued to the user more accurately, and the user can be reminded to intervene in the state of the vehicle compartment in time. Thus, it can prevent gravel, soil, or domestic garbage from being scattered on the road, causing damage to the road environment or traffic jams. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a flow chart of a method for detecting a carriage state according to an embodiment of the present invention;

[0051] Figure 2 A flowchart of a method for detecting a vehicle state according to another embodiment of the present invention for obtaining features of a current frame image;

[0052] Figure 3 A flowchart of a method for detecting a vehicle state according to another embodiment of the present invention for obtaining features of a current frame image;

[0053] Figure 4 A flowchart of a method for detecting a carriage state according to another embodiment of the present invention for obtaining a carriage state;

[0054] Figure 5 A flowchart of a method for detecting a carriage state according to another embodiment of the present invention for obtaining a carriage state;

[0055] Figure 6 A flowchart of determining the state of a carriage in a method for detecting the state of a carriage according to another embodiment of the present invention;

[0056] Figure 7 It is a schematic diagram of a vehicle compartment state detection device according to another embodiment of the present invention;

[0057] Figure 8 This is a schematic diagram of a feature acquisition submodule of a vehicle state detection device according to another embodiment of the present invention;

[0058] Fig. 9 This is a schematic diagram of a feature acquisition submodule of a vehicle state detection device according to another embodiment of the present invention;

[0059] Fig.10 This is a schematic diagram of a state acquisition submodule of a vehicle state detection device according to another embodiment of the present invention;

[0060] Fig.11 It is a schematic diagram of a state acquisition submodule of a vehicle state detection device according to another embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] The present invention provides a method for detecting a carriage state. Figure 1 As shown, including:

[0063] Step 100, obtaining a current frame image of the top of the carriage;

[0064] In some embodiments, the current frame image is captured by a vehicle-mounted camera mounted behind the front of the vehicle and facing the rear of the vehicle, and the shooting angle is a bird's-eye view of the transport vehicle compartment.

[0065] Step 200, using a convolutional or fully connected network to obtain features of the current frame image;

[0066] In some embodiments, a convolutional or fully connected network is used to process the current frame image. For example, the cargo, cover, and compartment of the current frame image can be segmented.

[0067] Step 310, determining the closed state of the compartment according to the proportion of the covering in the current frame image to the current frame image;

[0068] In some embodiments, a branch is used to output the degree of airtightness of the compartment, and a floating point number of 0 to 1 can be output, with 0 representing unsealed, 1 representing fully sealed, and intermediate values ​​representing intermediate sealed states. Generally, within the area enclosed by the compartment, the larger the area occupied by the covering, the higher the current degree of airtightness of the compartment. When the covering occupies the entire area, the output result is 1, and when there is no covering in the area, the output result is 0.

[0069] Step 320, determining the cargo state of the carriage according to the proportion of the carriage in the current frame image;

[0070] In some embodiments, a branch is used to output the degree of cargo loading in the carriage, and a floating point number of 0 to 1 may be output, with 0 representing no cargo loading, 1 representing full cargo loading, and intermediate values ​​representing intermediate cargo loading states. Generally, within the area enclosed by the carriage, the greater the proportion of the image occupied by the carriage, the lower the cargo loading degree of the current carriage. When the carriage occupies the entire area, it indicates that there is currently no cargo loading, and when the area occupied by the carriage is 0, it indicates that the current cargo loading state is fully loaded.

[0071] Step 400: Send corresponding alarm information according to the closed state and the cargo state.

[0072] In some embodiments, if the transport vehicle is in motion, the compartment is judged to be not sealed or semi-sealed, and the compartment is judged to be not empty, the monitoring device will send out an alarm signal to promptly remind the driver that there is cargo in the compartment but the compartment is not completely sealed and needs to be sealed, thereby preventing cargo such as gravel, soil, and domestic garbage from being scattered on the road, damaging the road environment, and blocking road traffic.

[0073] In the technical solution provided by this embodiment, by making a more precise judgment on the closed state and cargo state of the vehicle, a warning message can be issued to the user more accurately, and the user can be reminded to intervene in the state of the vehicle compartment in a timely manner. Thus, it can prevent gravel, soil, or domestic garbage from being scattered on the road, causing damage to the road environment or traffic congestion.

[0074] As an optional implementation, Figure 2 As shown, step 200 uses a convolutional or fully connected network to obtain the features of the current frame image, including:

[0075] Step 210, using the YOLO detection network to obtain the carriage position in the current frame image, and segmenting the carriage position to obtain a to-be-processed area;

[0076] In some embodiments, the YOLO detection network is used to obtain the car position and segment the car position, thereby eliminating the influence of the area outside the car on the subsequent judgment results.

[0077] Step 220: Process the area to be processed using a convolutional or fully connected network to obtain features of the current frame image.

[0078] In some embodiments, the above-mentioned steps 100 to 400 are used to process the area to be processed. Since the area to be processed only includes the area inside the vehicle compartment and will not be disturbed by the image outside the vehicle compartment, a more accurate judgment result can be obtained.

[0079] As an optional implementation, Figure 3 As shown, step 200 uses a convolutional or fully connected network to obtain the features of the current frame image, including:

[0080] Step 230, encoding the current frame image to form an image to be processed;

[0081] In some embodiments, by encoding the current frame image, the current frame image is encoded from a native image to an image in RGB or YUV or other color spaces to facilitate subsequent processing of the image.

[0082] Step 240, performing convolution or full connection processing on the image to be processed to obtain the target category of the image to be processed, the confidence corresponding to the target category, and the target position corresponding to the target category;

[0083] In some embodiments, the target category, the confidence corresponding to the target category and the target position of the carriage are obtained from the image to be processed through a fully convolutional or fully connected network, and the image is segmented according to the circumscribed rectangle of the target position of the carriage. The target category refers to the classification result after the target in the image is identified. For example, by identifying the image of the carriage part with the background image, one of the target categories can be obtained as the carriage and the other as the background image; the confidence corresponding to the target category indicates the credibility of the target category judgment result at the target position of the carriage. The higher the confidence, the higher the probability that the image within the circumscribed rectangle of the target position of the carriage is the image of the carriage.

[0084] Step 250, determining a feature map to be processed according to the target category, confidence level, target position and the image to be processed;

[0085] In some embodiments, before determining the feature map to be processed, it is necessary to make a judgment based on the confidence level. When the target category is part of a car, and its confidence level is greater than a predetermined value, for example, greater than 90%, the car target position is mapped to the image formed by the code, and a fixed-size feature map to be processed is obtained by pooling the feature layer. For example, using the method of regional pooling, the position of the target is proportionally mapped to the last layer of the shared feature encoding, and then pooled into a feature map of a fixed size. When its confidence level is not greater than a predetermined value, it indicates that the judgment result of the area is inaccurate, and therefore, the image to be processed is discarded.

[0086] Step 260, performing convolution or full connection processing on the feature map to be processed to obtain a probability map corresponding to the target category and a classification feature map corresponding to the target category;

[0087] In some embodiments, the processing of the feature map to be processed is performed through two branch networks, one of which obtains different types of probability maps through a fully convolutional or fully connected network, which may include the vehicle cover, vehicle interior wall, cargo and background; the other branch obtains a classification feature map through a fully convolutional or fully connected network.

[0088] Step 270, determining a target feature map based on the probability map and the classification feature map;

[0089] In some embodiments, the probability map obtained in the above steps and the classification feature map are added or multiplied to obtain a target feature map.

[0090] Step 280, using a convolutional or fully connected network to process the target feature map to obtain the features of the current frame image.

[0091] In some embodiments, the target characteristic map is processed using the aforementioned processing method of steps 200 to 400, that is, the airtightness of the compartment and the cargo status of the compartment are obtained.

[0092] In this embodiment, the target circumscribed rectangle of the image is segmented to obtain the image area corresponding to the target, and then the target category and the corresponding confidence are judged, and the image area with low confidence is discarded, thereby avoiding the calculation of the low confidence image area, which not only avoids the waste of computing resources, but also improves the accuracy of the calculation results. In addition, since the neural network is used for calculation in the aforementioned process, the user only needs to ensure that the captured image is input into the neural network model, which simplifies the method of use and reduces the difficulty of use.

[0093] As an optional implementation, Figure 4 As shown, determining the loading state and / or closing state of the carriage includes:

[0094] Step 330, using a single-input multi-output neural network model to output confidence values ​​of multiple degree intervals, wherein the degree intervals represent the degree of the cargo loading state and / or the closed state of the compartment;

[0095] In some embodiments, the current frame image is input into the neural network model, and the neural network model will output confidence values ​​of multiple degree intervals. For example, multiple branches are used to output the degree of airtightness of the carriage. The neural network model has 10 branches, and the branches range from 1 to 10, respectively, representing unsealed to fully sealed. Each branch can output a floating point number from 0 to 1, and the sum is 1. Then the degrees of airtightness of the 10 branches are weighted and summed to obtain the final degree of airtightness. If the final average output is 0.5, it means that the current state is semi-sealed. For another example, multiple branches are used to output the degree of cargo loading in the carriage. The neural network model has 10 branches, and the branches range from 1 to 10, respectively, representing unloaded to fully loaded. Each branch can output a floating point number from 0 to 1, and the sum is 1. Then the degrees of cargo loading of the 10 branches are weighted and summed to obtain the final degree of cargo loading. If the final average output is 0.5, it means that the current state is semi-loaded. The calculation process is illustrated below with specific numerical values. The intervals of the 10 branch outputs include ten intervals: 0-10%...90%-100%. The neural network will output a confidence value for each interval. For example, the confidence value in 0-10% is 0.5, the confidence value in 10%-20% is 0.5, and the others are 0. By weighted summing the ten intervals, we can get the loading state / closed state of the carriage as 0.5*10%+0.5*20%=15%, which means that the current loading state / closed state output of the carriage is 0.15, which is close to the unloaded state / unclosed state.

[0096] Step 340 , performing weighted summation on the multiple degree intervals as weight values ​​to obtain the cargo loading state and / or closed state of the carriage.

[0097] In some embodiments, a weighted sum is performed on the aforementioned multiple branches, and the weighted value can more accurately represent the loading state and / or closing state of the compartment.

[0098] In this embodiment, the neural network outputs the confidence values ​​for multiple cargo state interval values ​​or closed state interval values. After obtaining the confidence values ​​in each interval, the confidence values ​​are weighted and summed to obtain their weighted values. In the process of obtaining the confidence values, they are formed separately for multiple intervals, and there is no interference of values ​​between each other, which avoids the superposition of errors. In the process of weighted calculation, multiple interval values ​​are all involved in the calculation, which can ensure that the final weighted value is more accurate.

[0099] As an optional implementation, Figure 5 As shown, the carriage state detection method also includes:

[0100] Step 500, obtaining a loading state and / or a sealing state corresponding to a plurality of frames of images;

[0101] In some embodiments, after acquiring the closed state or loaded state of the vehicle compartment represented by a single frame image, the closed state or loaded state of the historical frames is acquired, for example, the closed state or loaded state within a specified number of frames, such as 30 frames, is acquired.

[0102] Step 600, obtaining the number of images having the same loading state and / or closing state;

[0103] In some embodiments, historical frame images of different sealing states and different loading states are counted. For example, historical frame images without cargo are counted to obtain the number of historical frame images without cargo, and unsealed historical frame images are counted to obtain the number of unsealed historical frame images, etc.

[0104] Step 700: When the number of images is greater than the predetermined threshold, the loading state and / or the closing state of the compartment is determined according to the loading state and / or the closing state corresponding to the number of images.

[0105] In some embodiments, if the maximum count value of a certain loading state and / or sealing state is greater than a set threshold value such as 20 frames, the corresponding state is output, otherwise an unknown state is output. Figure 6 Take 30 frames as an example to illustrate the output process of the cargo state of the carriage: obtain the cargo state of the carriage for multiple frames of images, such as 30 frames of images, respectively, and count the cargo state of the multiple frames of images. When multiple frames of images have the same cargo state, such as 0.5, determine the number of images under the cargo state. When the number of images exceeds the set threshold, such as 20 frames, the cargo state of the carriage is determined to be 0.5; when the number of images does not exceed the set threshold, such as 20 frames, the cargo state of the carriage is determined to be unknown. Similarly, continue with Figure 6 Take 30 frames as an example to illustrate the output process of the closed state of the car: the closed state of the car is obtained for multiple frames of images, such as 30 frames, and the closed state of the multiple frames is counted. When the multiple frames have the same closed state, such as 0.5, the number of images in the closed state is judged. When the number of images exceeds the set threshold, such as 20 frames, the closed state of the car is determined to be 0.5; when the number of images does not exceed the set threshold, such as 20 frames, the closed state of the car is determined to be unknown.

[0106] In this embodiment, the judgment results of multiple frames of images are statistically analyzed, and the final loading state and / or sealing state of the compartment is determined based on the statistical results. Multiple frames of images are calculated separately without interfering with each other, that is, multiple frames of images obtain independent calculation results, and then the final loading state and / or sealing state is determined based on the statistical results of the loading state and / or sealing state of the multiple frames of images. In this way, errors in the calculation of a single frame of images that lead to errors in the judgment of the loading state and / or sealing state are avoided, which is conducive to improving the accuracy of the judgment results.

[0107] The embodiment of the present invention provides a vehicle compartment state detection device, such as Figure 7 As shown, including:

[0108] The vehicle-mounted camera is set at the rear of the vehicle head and is used to obtain the current frame image of the top of the vehicle compartment;

[0109] In some embodiments, the current frame image is captured by a vehicle-mounted camera mounted behind the front of the vehicle and facing the rear of the vehicle, and the shooting angle is a bird's-eye view of the transport vehicle compartment.

[0110] The algorithm operation module is connected to the vehicle camera in communication, wherein the algorithm operation module includes: a feature acquisition submodule, which is used to obtain the features of the current frame image by using a convolution or fully connected network; in some embodiments, the current frame image is photographed by a vehicle camera set up behind the front of the vehicle and facing the rear of the vehicle, and the shooting angle is to shoot the carriage of the transport vehicle from above. A state acquisition submodule is used to determine the closed state of the carriage according to the proportion of the current frame image occupied by the covering in the current frame image; in some embodiments, a branch is used to output the degree of sealing of the carriage, and a floating point number of 0 to 1 can be output, with 0 representing unsealed, 1 representing fully sealed, and the intermediate value representing the intermediate sealed state. Generally, within the area enclosed by the carriage, the larger the area occupied by the covering, the higher the current degree of sealing of the carriage. When the covering occupies the entire area, the output result is 1, and when there is no covering in the area, the output result is 0. According to the proportion of the carriage in the current frame image occupied by the current frame image, the cargo state of the carriage is determined. In some embodiments, a branch is used to output the degree of cargo loading in the carriage, and a floating point number of 0 to 1 may be output, with 0 representing no cargo loading, 1 representing full cargo loading, and intermediate values ​​representing intermediate cargo loading states. Generally, within the area enclosed by the carriage, the greater the proportion of the image occupied by the carriage, the lower the cargo loading degree of the current carriage. When the carriage occupies the entire area, it indicates that there is currently no cargo loading, and when the area occupied by the carriage is 0, it indicates that the current cargo loading state is fully loaded.

[0111] The alarm module is used to send corresponding alarm information according to the closed state and the cargo state.

[0112] In some embodiments, if the transport vehicle is in motion, the compartment is judged to be not sealed or semi-sealed, and the compartment is judged to be not empty, the monitoring device will send out an alarm signal to promptly remind the driver that there is cargo in the compartment but the compartment is not completely sealed and needs to be sealed, thereby preventing cargo such as gravel, soil, and domestic garbage from being scattered on the road, damaging the road environment, and blocking road traffic.

[0113] In the technical solution provided by this embodiment, by making a more precise judgment on the closed state and cargo state of the vehicle, a warning message can be issued to the user more accurately, and the user can be reminded to intervene in the state of the vehicle compartment in a timely manner. Thus, it can prevent gravel, soil, or domestic garbage from being scattered on the road, causing damage to the road environment or traffic congestion.

[0114] As an optional implementation, Figure 8 As shown, the feature acquisition submodule includes:

[0115] A segmentation unit, used for acquiring the carriage position in the current frame image by using a YOLO detection network, and segmenting the carriage position to obtain a to-be-processed area;

[0116] In some embodiments, the YOLO detection network is used to obtain the car position and segment the car position, thereby eliminating the influence of the area outside the car on the subsequent judgment results.

[0117] The feature acquisition unit is used to process the area to be processed by using a convolutional or fully connected network to obtain the features of the current frame image.

[0118] In some embodiments, the above-mentioned steps 100 to 400 are used to process the area to be processed. Since the area to be processed only includes the area inside the vehicle compartment and will not be disturbed by the image outside the vehicle compartment, a more accurate judgment result can be obtained.

[0119] As an optional implementation, Fig. 9 As shown, the feature acquisition submodule includes:

[0120] An encoding unit, used for encoding the current frame image to form an image to be processed;

[0121] In some embodiments, by encoding the current frame image, the current frame image is encoded from a native image to an image in RGB or YUV or other color spaces to facilitate subsequent processing of the image.

[0122] an identification unit, configured to perform convolution or full connection processing on the image to be processed to obtain a target category of the image to be processed, a confidence level corresponding to the target category, and a target position corresponding to the target category;

[0123] In some embodiments, the target category, the confidence corresponding to the target category and the target position of the carriage are obtained from the image to be processed through a fully convolutional or fully connected network, and the image is segmented according to the circumscribed rectangle of the target position of the carriage. The target category refers to the classification result after the target in the image is identified. For example, by identifying the image of the carriage part with the background image, one of the target categories can be obtained as the carriage and the other as the background image; the confidence corresponding to the target category indicates the credibility of the target category judgment result at the target position of the carriage. The higher the confidence, the higher the probability that the image within the circumscribed rectangle of the target position of the carriage is the image of the carriage.

[0124] A pooling unit, used to determine a feature map to be processed according to the target category, confidence, target position and image to be processed;

[0125] In some embodiments, before determining the feature map to be processed, it is necessary to make a judgment based on the confidence level. When the target category is part of a car, and its confidence level is greater than a predetermined value, for example, greater than 90%, the car target position is mapped to the image formed by the code, and a fixed-size feature map to be processed is obtained by pooling the feature layer. For example, using the method of regional pooling, the position of the target is proportionally mapped to the last layer of the shared feature encoding, and then pooled into a feature map of a fixed size. When its confidence level is not greater than a predetermined value, it indicates that the judgment result of the area is inaccurate, and therefore, the image to be processed is discarded.

[0126] A classification unit, used for performing convolution or full connection processing on the feature map to be processed to obtain a probability map corresponding to the target category and a classification feature map corresponding to the target category;

[0127] In some embodiments, the processing of the feature map to be processed is performed through two branch networks, one of which obtains different types of probability maps through a fully convolutional or fully connected network, which may include the vehicle cover, vehicle interior wall, cargo and background; the other branch obtains a classification feature map through a fully convolutional or fully connected network.

[0128] A target unit, used to determine a target feature map based on the probability map and the classification feature map;

[0129] In some embodiments, the probability map obtained in the above steps and the classification feature map are added or multiplied to obtain a target feature map.

[0130] A feature acquisition unit is used to process the target feature map using a convolutional or fully connected network to acquire the features of the current frame image.

[0131] In some embodiments, the target characteristic map is processed using the aforementioned processing method of steps 200 to 400, that is, the airtightness of the compartment and the cargo status of the compartment are obtained.

[0132] In this embodiment, the target circumscribed rectangle of the image is segmented to obtain the image area corresponding to the target, and then the target category and the corresponding confidence are judged, and the image area with low confidence is discarded, thereby avoiding the calculation of the low confidence image area, which not only avoids the waste of computing resources, but also improves the accuracy of the calculation results. In addition, since the neural network is used for calculation in the aforementioned process, the user only needs to ensure that the captured image is input into the neural network model, which simplifies the method of use and reduces the difficulty of use.

[0133] As an optional implementation, Fig.10 As shown, the status acquisition submodule includes:

[0134] A multi-output unit, used to output confidence values ​​of multiple degree intervals using a neural network model, wherein the degree intervals represent the degree of the cargo state and / or the closed state of the compartment;

[0135] In some embodiments, the current frame image is input into a single-input multi-output neural network model, and the neural network model will output multiple values. For example, multiple branches are used to output the degree of airtightness of the carriage. The neural network model has 10 branches, and the branches range from 1 to 10, respectively, representing unsealed to fully sealed. Each branch can output a floating point number from 0 to 1, and the sum is 1. Then the degrees of airtightness of the 10 branches are weighted and summed to obtain the final degree of airtightness. If the final average output is 0.5, it means that the current state is semi-sealed. For another example, multiple branches are used to output the degree of cargo loading in the carriage. The neural network model has 10 branches, and the branches range from 1 to 10, respectively, representing unloaded to fully loaded. Each branch can output a floating point number from 0 to 1, and the sum is 1. Then the degrees of cargo loading of the 10 branches are weighted and summed to obtain the final degree of cargo loading. If the final average output is 0.5, it means that the current state is semi-loaded. The calculation process is illustrated below with specific numerical values. The intervals of the 10 branch outputs include ten intervals: 0-10%...90%-100%. The neural network will output a confidence value for each interval. For example, the confidence value in 0-10% is 0.5, the confidence value in 10%-20% is 0.5, and the others are 0. By weighted summing the ten intervals, we can get the loading state / closed state of the carriage as 0.5*10%+0.5*20%=15%, which means that the current loading state / closed state output of the carriage is 0.15, which is close to the unloaded state / unclosed state.

[0136] The averaging unit is used to perform weighted summation on the multiple values ​​using the degree interval as the weight value to obtain the cargo loading state and / or the closed state of the carriage.

[0137] In some embodiments, a weighted sum is performed on the aforementioned multiple branches, and the weighted value can more accurately represent the loading state and / or closing state of the compartment.

[0138] In this embodiment, the neural network outputs the confidence values ​​for multiple cargo state interval values ​​or closed state interval values. After obtaining the confidence values ​​in each interval, the confidence values ​​are weighted and summed to obtain their weighted values. In the process of obtaining the confidence values, they are formed separately for multiple intervals, and there is no interference of values ​​between each other, which avoids the superposition of errors. In the process of weighted calculation, multiple interval values ​​are all involved in the calculation, which can ensure that the final weighted value is more accurate.

[0139] As an optional implementation, Fig.11 As shown, it also includes:

[0140] A data collection module, used to obtain a loading state and / or a closing state corresponding to a plurality of frames of images;

[0141] In some embodiments, after acquiring the closed state or loaded state of the vehicle compartment represented by a single frame image, the closed state or loaded state of the historical frames is acquired, for example, the closed state or loaded state within a specified number of frames, such as 30 frames, is acquired.

[0142] A data statistics module, used for obtaining the number of images with the same loading state and / or closed state;

[0143] In some embodiments, historical frame images of different sealing states and different loading states are counted. For example, historical frame images without cargo are counted to obtain the number of historical frame images without cargo, and unsealed historical frame images are counted to obtain the number of unsealed historical frame images, etc.

[0144] A state determination module is used to determine the loading state and / or closing state of the compartment according to the loading state and / or closing state corresponding to the number of images when the number of images is greater than the predetermined threshold.

[0145] In some embodiments, if the maximum count value of a certain closed state is greater than a set threshold value such as 20 frames, the corresponding state is output, otherwise an unknown state is output. Figure 6Take 30 frames as an example to illustrate the output process of the cargo state of the carriage: obtain the cargo state of the carriage for multiple frames of images, such as 30 frames of images, respectively, and count the cargo state of the multiple frames of images. When multiple frames of images have the same cargo state, such as 0.5, determine the number of images under the cargo state. When the number of images exceeds the set threshold, such as 20 frames, the cargo state of the carriage is determined to be 0.5; when the number of images does not exceed the set threshold, such as 20 frames, the cargo state of the carriage is determined to be unknown. Similarly, continue with Figure 6 Take 30 frames as an example to illustrate the output process of the closed state of the car: the closed state of the car is obtained for multiple frames of images, such as 30 frames, and the closed state of the multiple frames is counted. When the multiple frames have the same closed state, such as 0.5, the number of images in the closed state is judged. When the number of images exceeds the set threshold, such as 20 frames, the closed state of the car is determined to be 0.5; when the number of images does not exceed the set threshold, such as 20 frames, the closed state of the car is determined to be unknown.

[0146] In this embodiment, the judgment results of multiple frames of images are statistically analyzed, and the final loading state and / or sealing state of the compartment is determined based on the statistical results. Multiple frames of images are calculated separately without interfering with each other, that is, multiple frames of images obtain independent calculation results, and then the final loading state and / or sealing state is determined based on the statistical results of the loading state and / or sealing state of the multiple frames of images. In this way, errors in the calculation of a single frame of images that lead to errors in the judgment of the loading state and / or sealing state are avoided, which is conducive to improving the accuracy of the judgment results.

[0147] In the above-mentioned embodiments, the algorithm operation platform may be an embedded algorithm operation platform. The alarm module may be an audible and visual alarm, which is used to broadcast and remind the driver of illegal or dangerous transportation behaviors on the spot.

[0148] Those skilled in the art should be able to understand that the carriage state detection device in the above embodiments may further include:

[0149] Communicator, uses mobile signals to upload detection information to the background.

[0150] The display screen is used to display the real-time video collected by the vehicle camera, alarm prompt images, and touch control images.

[0151] Those skilled in the art can understand that all or part of the processes in the above method embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes in the above method embodiments. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0152] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for detecting a carriage state, characterized in that: include: Get the current frame image of the top of the carriage; Using a convolutional or fully connected network to obtain features of the current frame image; Determining the closed state of the compartment according to the proportion of the covering in the current frame image to the current frame image; Determining the cargo state of the carriage according to the proportion of the carriage in the current frame image; Sending corresponding alarm information according to the closed state and the cargo state; Wherein, using a convolutional or fully connected network to obtain the features of the current frame image includes: Encoding the current frame image to form an image to be processed; Performing convolution or full connection processing on the image to be processed to obtain a target category of the image to be processed, a confidence level corresponding to the target category, and a target position corresponding to the target category; Determining a feature map to be processed according to the target category, confidence level, target position, and image to be processed; Performing convolution or full connection processing on the feature map to be processed to obtain a probability map corresponding to the target category and a classification feature map corresponding to the target category; Determine a target feature map based on the probability map and the classification feature map; The target feature map is processed using a convolutional or fully connected network to obtain the features of the current frame image.

2. The method according to claim 1, characterized in that The features of the current frame image are obtained by using a convolutional or fully connected network, including: The YOLO detection network is used to obtain the carriage position in the current frame image, and the carriage position is segmented to obtain the area to be processed; The area to be processed is processed using a convolutional or fully connected network to obtain features of the current frame image.

3. The method according to claim 1, characterized in that Determining the loading state and / or the closing state of the carriage includes: A neural network model is used to output confidence values ​​of a plurality of degree intervals, wherein the degree intervals represent the degree of the cargo loading state and / or the closed state of the compartment; The multiple values ​​are weighted and summed using the degree interval as the weight value to obtain the cargo loading state and / or the closed state of the vehicle compartment.

4. The method according to claim 1, characterized in that: Also includes: Acquire the loading state and / or closing state corresponding to the multiple frames of images; Obtaining the number of images having the same loading state and / or closure state; When the number of the images is greater than a predetermined threshold, the loading state and / or the closing state of the compartment is determined according to the loading state and / or the closing state corresponding to the number of the images.

5. A carriage state detection device, characterized in that: include: The vehicle-mounted camera is set at the rear of the vehicle head and is used to obtain the current frame image of the top of the vehicle compartment; an algorithm operation module, which is in communication connection with the vehicle-mounted camera, wherein the algorithm operation module includes: a feature acquisition submodule, which is used to acquire the features of the current frame image by using a convolutional or fully connected network; a state acquisition submodule, which is used to determine the closed state of the compartment according to the proportion of the covering in the current frame image to the current frame image; and to determine the cargo state of the compartment according to the proportion of the compartment in the current frame image to the current frame image; An alarm module, used for sending corresponding alarm information according to the closed state and the cargo state; Among them, the feature acquisition submodule includes: An encoding unit, used for encoding the current frame image to form an image to be processed; an identification unit, configured to perform convolution or full connection processing on the image to be processed to obtain a target category of the image to be processed, a confidence level corresponding to the target category, and a target position corresponding to the target category; A pooling unit, used to determine a feature map to be processed according to the target category, confidence, target position and image to be processed; A classification unit, used for performing convolution or full connection processing on the feature map to be processed to obtain a probability map corresponding to the target category and a classification feature map corresponding to the target category; A target unit, used to determine a target feature map based on the probability map and the classification feature map; A feature acquisition unit is used to process the target feature map using a convolutional or fully connected network to acquire the features of the current frame image.

6. The device according to claim 5, characterized in that The feature acquisition submodule includes: A segmentation unit, used for acquiring the carriage position in the current frame image by using a YOLO detection network, and segmenting the carriage position to obtain a to-be-processed area; The feature acquisition unit is used to process the area to be processed by using a convolutional or fully connected network to obtain the features of the current frame image.

7. The device according to claim 5, characterized in that in, The state acquisition submodule includes: A multi-output unit, used to output confidence values ​​of multiple degree intervals using a neural network model, wherein the degree intervals represent the degree of the cargo state and / or the closed state of the compartment; The averaging unit is used to perform weighted summation on the multiple values ​​using the degree interval as the weight value to obtain the cargo loading state and / or the closed state of the carriage.

8. The device according to claim 5, characterized in that Also includes: A data collection module, used to obtain a loading state and / or a closed state corresponding to a plurality of frames of images; A data statistics module, used for obtaining the number of images with the same loading state and / or closed state; The state determination module is used to determine the loading state and / or the closing state of the compartment according to the loading state and / or the closing state corresponding to the number of images when the number of images is greater than a predetermined threshold.

Images