Vehicle state identification method, device and equipment and vehicle

Abstract

The embodiment of the invention provides a vehicle state recognition method, device and equipment and a vehicle. The method comprises the steps that firstly, when it is detected that a vehicle enters a detection area, at least one first image of the vehicle is obtained, then the target loading state of the vehicle is determined according to the at least one first image and a first model, and the target loading state comprises a first full-load state or a second non-full-load state; the first model is obtained by training based on a plurality of first historical images and the labeled loading state of each first historical image. According to the method, the vehicle state is recognized, and the accuracy and efficiency of vehicle state recognition are effectively improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus, device, and vehicle for recognizing vehicle status. Background Technology

[0002] Wide-body mining vehicles are typically used in mining operations, such as transporting ore and waste. Due to the harsh working environment and heavy transportation tasks in mines, it is essential to identify the full load status of vehicles in order to ensure operational efficiency and safety.

[0003] Traditional methods for identifying the full load status of vehicles mainly rely on manual identification to determine the full load status of wide-body mining vehicles, which involves visually inspecting the vehicles to determine their full load status.

[0004] However, existing methods for identifying fully loaded vehicles suffer from low accuracy and efficiency. Summary of the Invention

[0005] This application provides a method, apparatus, device, and vehicle for identifying vehicle status, in order to improve the accuracy and efficiency of vehicle status identification.

[0006] In a first aspect, embodiments of this application provide a method for identifying vehicle status, including:

[0007] When a vehicle is detected entering the detection area, at least one first image of the vehicle is acquired;

[0008] Based on the at least one first image and the first model, the target load state of the vehicle is determined, the target load state including: a first state of being fully loaded or a second state of being partially loaded, the first model being trained based on multiple first historical images and the labeled load state of each first historical image.

[0009] In one or more embodiments, determining the target load state of the vehicle based on the at least one first image and the first model includes:

[0010] The first image is input into the second model to obtain the second image with the highest quality score among the first images. The second model is trained based on the first image and the labeled quality score of each second historical image.

[0011] The second image is input into the first model to obtain the target load state of the vehicle.

[0012] In one or more embodiments, before inputting the second image into the first model to obtain the target load state of the vehicle, the method further includes:

[0013] S1, if the quality score of the second image is less than the preset score, obtain at least one new first image from the detection area;

[0014] S2, input the new at least one first image into the second model to obtain a new second image with the highest quality score among the new at least one first image;

[0015] S3. If the quality score of the new second image is greater than or equal to the preset score, then the new second image is used as the second image input to the first model. Otherwise, repeat steps S1-S3 until the quality score of the new second image is greater than or equal to the preset score.

[0016] In one or more embodiments, the method further includes:

[0017] If the target load status indicates that the vehicle is in a second state of not being fully loaded, mark the unloaded area indicated by the second state and the corresponding confidence level data on the second image.

[0018] In one or more embodiments, before determining the target load state of the vehicle based on the at least one first image and the first model, the method further includes:

[0019] S4, input multiple first historical images into the YOLOv5s model to obtain the predicted object state corresponding to each first historical image. The feature fusion Neck layer in the YOLOv5s model includes a cross-stage feature fusion C3 module, and the spatial slice channel reorganization Focus module is replaced with a 6×6 convolution.

[0020] S5, determine the loss value based on the unloaded area indicated by each predicted cargo status and the unloaded area indicated by the labeled cargo status of each first historical image;

[0021] S6. Based on the loss value, adjust the parameters in the YOLOv5s model and repeat S4-S6 until the new loss value converges, and determine the YOLOv5s model as the first model.

[0022] In one or more embodiments, before inputting the at least one first image into the second model to obtain the second image with the highest quality score among the at least one first image, the method further includes:

[0023] The second model is obtained by training a residual neural network ResNet-18 based on at least one second historical image and the labeled quality score of each second historical image.

[0024] In one or more embodiments, before inputting multiple first historical images into the YOLOv5s model to obtain the predicted object state corresponding to each first historical image, the method further includes:

[0025] The plurality of first historical images are preprocessed, including: size conversion, three primary colors processing, and pixel filling.

[0026] Secondly, embodiments of this application provide a vehicle status identification device, comprising:

[0027] The acquisition module is used to acquire at least one first image of the vehicle when it is detected that the vehicle has entered the detection area;

[0028] The determination module determines the target load state of the vehicle based on the at least one first image and the first model. The target load state includes either a first state of being fully loaded or a second state of being partially loaded. The first model is trained based on multiple first historical images and the labeled load state of each first historical image.

[0029] In one or more embodiments, the determining module is specifically used for:

[0030] The at least one first image is input into the second model to obtain the second image with the highest quality score among the at least one first image. The second model is trained based on at least one second historical image and the labeled quality score of each second historical image.

[0031] The second image is input into the first model to obtain the target load state of the vehicle.

[0032] In one or more embodiments, before inputting the second image into the first model to obtain the target load state of the vehicle, the determining module is further configured to:

[0033] S1, if the quality score of the second image is less than the preset score, obtain at least one new first image from the detection area;

[0034] S2, input the new at least one first image into the second model to obtain a new second image with the highest quality score among the new at least one first image;

[0035] S3. If the quality score of the new second image is greater than or equal to the preset score, then the new second image is used as the second image input to the first model. Otherwise, repeat steps S1-S3 until the quality score of the new second image is greater than or equal to the preset score.

[0036] In one or more embodiments, the determining module is further configured to:

[0037] If the target load status indicates that the vehicle is in a second state of not being fully loaded, mark the unloaded area indicated by the second state and the corresponding confidence level data on the second image.

[0038] In one or more embodiments, before determining the target load state of the vehicle based on the at least one first image and the first model, the determining module is further configured to:

[0039] S4, input multiple first historical images into the YOLOv5s model to obtain the predicted object state corresponding to each first historical image. The feature fusion Neck layer in the YOLOv5s model includes a cross-stage feature fusion C3 module, and the spatial slice channel reorganization Focus module is replaced with a 6×6 convolution.

[0040] S5, determine the loss value based on the unloaded area indicated by each predicted cargo status and the unloaded area indicated by the labeled cargo status of each first historical image;

[0041] S6. Based on the loss value, adjust the parameters in the YOLOv5s model and repeat S4-S6 until the new loss value converges, and determine the YOLOv5s model as the first model.

[0042] In one or more embodiments, before inputting the at least one first image into the second model to obtain the second image with the highest quality score among the at least one first image, the determining module is further configured to:

[0043] The second model is obtained by training a residual neural network ResNet-18 based on at least one second historical image and the labeled quality score of each second historical image.

[0044] In one or more embodiments, before inputting multiple first historical images into the YOLOv5s model to obtain the predicted object state corresponding to each first historical image, the determining module is further configured to:

[0045] The plurality of first historical images are preprocessed, including: size conversion, three primary colors processing, and pixel filling.

[0046] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0047] The memory stores computer-executed instructions;

[0048] The processor executes computer execution instructions stored in the memory, such that the processor, when executed, is used to implement the method described in the first aspect and any of the embodiments above.

[0049] Fourthly, embodiments of this application provide an engineering vehicle for performing the methods described in the first aspect and any of the embodiments above, or including the electronic equipment described in the third aspect.

[0050] Fifthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods described in the first aspect and any of the embodiments above.

[0051] In a sixth aspect, this application provides a computer program product, including a computer program that, when executed by a processor, is used to implement a vehicle state identification method as described in the first aspect and various possible implementations of the first aspect.

[0052] The vehicle state recognition method, apparatus, device, and vehicle provided in this application embodiment first acquire at least one first image of the vehicle when a vehicle is detected entering a detection area. Then, based on the at least one first image and a first model, the target load state of the vehicle is determined. The target load state includes either a fully loaded first state or a partially loaded second state. The first model is trained based on multiple first historical images and the labeled load state of each first historical image. In this method, by acquiring at least one first image of the vehicle when it is detected entering the detection area, the target load state of the vehicle can be automatically identified and determined based on the at least one first image and the first model, eliminating the need for manual operation and effectively improving the efficiency of vehicle state recognition. By training the first model based on multiple first historical images and the labeled load state of each first historical image, the first model can learn the feature information of different first historical images, determine the load state of the vehicle corresponding to different first historical images, and optimize the first model by combining the labeled load state of each first historical image, thereby effectively improving the accuracy of vehicle state recognition. Attached Figure Description

[0053] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0054] Figure 1 Flowchart of the vehicle status identification method provided in the embodiments of this application Figure 1 ;

[0055] Figure 2Flowchart of the vehicle status identification method provided in the embodiments of this application Figure 2 ;

[0056] Figure 3 Flowchart of the vehicle status identification method provided in the embodiments of this application Figure 3 ;

[0057] Figure 4 A schematic diagram of the structure of the vehicle status recognition device provided in the embodiments of this application;

[0058] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0059] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0060] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0061] Before introducing the embodiments of this application, the application background of the embodiments of this application will be explained first:

[0062] Wide-body mining vehicles are typically used in mining operations, such as transporting ore and waste. Due to the harsh working environment and heavy transportation tasks in mines, it is essential to identify the full load status of vehicles in order to ensure operational efficiency and safety.

[0063] Traditional methods for identifying the full load status of vehicles mainly rely on manual identification to determine the full load status of wide-body mining vehicles, which involves visually inspecting the vehicles to determine their full load status.

[0064] However, existing methods for identifying fully loaded vehicles suffer from low accuracy and efficiency.

[0065] The vehicle status identification method provided in this application aims to solve the above-mentioned technical problems of the prior art. The inventive concept of this application is as follows: Traditional vehicle full-load status identification methods have low accuracy and efficiency due to manual identification. Identifying the full-load status of a vehicle mainly involves observing whether the cargo area in the vehicle image is completely filled, thereby determining whether the vehicle is fully loaded or not. If the vehicle image can be effectively analyzed, the vehicle's cargo status can be accurately and quickly determined. Therefore, this application first acquires at least one first image of the vehicle when a vehicle is detected entering the detection area. Subsequent analysis and processing can be performed based on the at least one first image. A first model is trained based on multiple first historical images and the labeled cargo status of each first historical image. According to the at least one first image and the first model, the first model effectively analyzes the at least one first image to determine whether the cargo area in the at least one first image is completely filled. This can automatically identify and determine the target cargo status of the vehicle without manual operation, improving the accuracy and efficiency of vehicle status identification.

[0066] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0067] Figure 1 Flowchart of the vehicle status identification method provided in the embodiments of this application Figure 1 .like Figure 1 As shown, the method for identifying the vehicle's status includes the following steps:

[0068] S110. When a vehicle is detected entering the detection area, acquire at least one first image of the vehicle.

[0069] In this step, a detection area is pre-set. In order to identify the cargo status of the vehicle, when the vehicle is detected to enter the detection area, at least one first image of the vehicle can be acquired as the basis for subsequent analysis of the cargo status of the vehicle.

[0070] In one possible implementation, a fixed camera group can be set up in the detection area to capture at least one first image of the vehicle. The camera group includes a camera unit, a target recognition unit (TRU), and a millimeter-wave radar unit.

[0071] For example, when the millimeter-wave radar unit detects that a vehicle has entered the detection area, the target recognition unit identifies and determines the specific location of the vehicle, and the camera unit captures at least one first image of the vehicle. The shooting parameters of the camera unit can be set as follows: the shooting resolution is 1920×1080, the shooting frame rate is 30fps, and the camera unit can support a low light enhancement mode, so that it can obtain a clearer image in low light environment.

[0072] In one possible implementation, after acquiring at least one first image of the vehicle, it can be preprocessed to remove irrelevant background from the at least one first image, retaining only the effective loading area of ​​the vehicle, and the at least one first image can be converted from RGB channel format to BGR channel format.

[0073] In addition, at least one first image can be uploaded to the cloud and stored in a categorized manner according to "mine area number - shooting date - vehicle number".

[0074] S120. Determine the target load status of the vehicle based on at least one first image and the first model;

[0075] The target loading state includes either a first state of being fully loaded or a second state of being partially loaded. The first model is trained based on multiple first historical images and the labeled loading state of each first historical image.

[0076] In this step, a first model can be obtained by training based on multiple first historical images and the labeled loading state of each first historical image. The loading state can include: a first state of being fully loaded or a second state of being not fully loaded. Then, at least one first image is input into the first model to determine the target loading state of the vehicle.

[0077] For example, the first state of being fully loaded means that the vehicle has reached or is close to its maximum carrying capacity, which is manifested as the vehicle compartment being full or nearly full, that is, the central area and the edges of the compartment are in a saturated state with no gaps.

[0078] The second state of being unloaded indicates that the vehicle has not reached its maximum carrying capacity. This is manifested in the fact that the vehicle's cargo compartment is not completely filled, that is, there are large gaps at the edges of the cargo compartment, or the cargo in the center of the cargo compartment is less than the normal height for a fully loaded cargo compartment.

[0079] In one possible implementation, multiple first historical images are input into a first model, which outputs the predicted object states of the multiple first historical images. The predicted object states are compared with the labeled object states of each first historical image, and the first model is trained and its parameters are updated.

[0080] The vehicle state recognition method provided in this application first acquires at least one first image of the vehicle when a vehicle is detected entering a detection area. Then, based on the at least one first image and a first model, the target load state of the vehicle is determined. The target load state includes either a fully loaded first state or a partially loaded second state. The first model is trained based on multiple first historical images and the labeled load states of each first historical image. In this embodiment, by acquiring at least one first image of the vehicle when it is detected entering the detection area, the target load state of the vehicle can be automatically identified and determined based on the at least one first image and the first model, without manual operation, effectively improving the efficiency of vehicle state recognition. By training the first model based on multiple first historical images and the labeled load states of each first historical image, the first model can learn the feature information of different first historical images, determine the load state of the vehicle corresponding to different first historical images, and optimize the first model by combining the labeled load states of each first historical image, thereby effectively improving the accuracy of vehicle state recognition.

[0081] Based on the above embodiments, Figure 2 Flowchart of the vehicle status identification method provided in the embodiments of this application Figure 2 .like Figure 2 As shown, a possible implementation of step S120 above also includes the following steps:

[0082] S210. Input at least one first image into the second model to obtain the second image with the highest quality score among at least one first image;

[0083] The second model is trained based on at least one second historical image and the labeled quality score of each second historical image.

[0084] In this step, a second model can be obtained by training based on at least one second historical image and the labeled quality score of each second historical image. By inputting at least one first image into the second model, the second image with the highest quality score among at least one first image can be obtained.

[0085] In one possible implementation, at least one second historical image is input into the second model, which can output a predicted quality score for at least one second historical image. The predicted quality score is compared with the labeled quality score of each second historical image, the second model is trained, and the model parameters of the second model are updated.

[0086] In one possible implementation, prior to step S210 above, the vehicle state recognition method further includes: training a residual neural network ResNet-18 based on at least one second historical image and the labeled quality score of each second historical image to obtain a second model.

[0087] For example, the quality score of each second historical image is obtained by scoring each second historical image in the range of 0-10 based on the quality requirements of the first model for the input image.

[0088] If the vehicle's cargo compartment is fully visible in the second historical image, and the area where the material is loaded occupies more than 60% of the second image, then the annotation quality score of the second historical image is 10 points.

[0089] If the vehicle compartment area in the second historical image is small, obscured, or blurry, then the annotation quality score of the second historical image is 0.

[0090] In addition, image enhancement can be performed on at least one second historical image, including adjusting the brightness of at least one second historical image by ±20% and the contrast by ±10% to simulate the complex lighting conditions in the mining area. Furthermore, at least one second historical image can be standardized by setting the mean and standard deviation of at least one second historical image to obtain at least one second historical image in a standard format.

[0091] In one possible implementation, the ResNet-18 residual neural network is trained based on at least one second historical image and the labeled quality score of each second historical image. This can be achieved by removing the classification layer from the ResNet-18 residual neural network and retaining only the feature extraction layer (which can output a 512-dimensional feature vector). The feature extraction layer extracts features from at least one second historical image to obtain a 512-dimensional feature vector. The 512-dimensional feature vector is then fused through a fully connected layer to obtain the predicted quality score corresponding to at least one second historical image.

[0092] Based on the predicted quality score corresponding to at least one second historical image and the labeled quality score of each second historical image, the mean squared error (MSE) loss function is used to adapt the probability labels of continuous quality scores and optimize the model parameters of the residual neural network ResNet-18.

[0093] S220. Input the second image into the first model to obtain the target load state of the vehicle.

[0094] In this step, the second image is used as input to the first model, which analyzes and processes the second image to determine the target load status of the vehicle corresponding to the second image.

[0095] In one possible implementation, prior to step S220 above, the vehicle state identification method further includes the following steps:

[0096] S1, if the quality score of the second image is less than the preset score, obtain at least one new first image from the detection area.

[0097] For example, a preset score is used to indicate the minimum threshold required to satisfy the quality score requirement of the first model for the input image.

[0098] In one possible implementation, if the quality score of the second image is less than the preset score, then the second image is a low-quality image and does not meet the quality requirements of the subsequent first model for the input image. Therefore, at least one new first image needs to be obtained from the detection area.

[0099] The new first image can be obtained by enhancing the image acquired in the detection area or by inpainting the image acquired in the detection area.

[0100] S2, input at least one new first image into the second model to obtain a new second image with the highest quality score among the at least one new first image.

[0101] For example, at least one new first image is used as input to the second model, which extracts features from the at least one new first image and outputs a quality score for the at least one new first image. The first image with the highest quality score is then used as the new second image with the highest quality score.

[0102] S3. If the quality score of the new second image is greater than or equal to the preset score, then the new second image is used as the second image input to the first model. Otherwise, repeat steps S1-S3 until the quality score of the new second image is greater than or equal to the preset score.

[0103] For example, if the quality score of the new second image is greater than or equal to the preset score, and meets the quality requirements of the first model for the input image, then the new second image can be used as the second image input to the first model.

[0104] If the quality score of the new second image is still lower than the preset score, then at least one new first image is obtained from the detection area again until the quality score of the new second image is greater than or equal to the preset score.

[0105] In one possible implementation, the vehicle status identification method further includes: if the target load status indicates that the vehicle is in a second state of not being fully loaded, marking the unloaded area indicated by the second state and the corresponding confidence data on the second image.

[0106] For example, the bounding box parameters of the unfilled area include: the coordinates of the bounding box center point (x_center, y_center), the width of the bounding box, and the height of the bounding box.

[0107] The confidence data corresponding to the unloaded region can indicate the probability that an unloaded second state exists within the bounding box. The value ranges from 0 to 1. The closer the confidence data is to 1, the higher the confidence of the unloaded region.

[0108] In one possible implementation, the second image is input into the first model to obtain the target load state of the vehicle. When the target load state indicates that the vehicle is in a second state where it is not fully loaded, the first model simultaneously outputs the bounding box parameters of the unloaded area indicated by the second state and the confidence data corresponding to the unloaded area. An image processing library (such as OpenCV) annotates the bounding box of the unloaded area on the second image based on the bounding box parameters of the unloaded area, and annotates the confidence data corresponding to the unloaded area on the bounding box.

[0109] The vehicle state recognition method provided in this application first inputs at least one first image into a second model to obtain a second image with the highest quality score among the at least one first image. The second model is trained based on at least one second historical image and the labeled quality score of each second historical image. Then, the second image is input into the first model to obtain the target load state of the vehicle. In this embodiment, by inputting at least one first image into the second model, the quality score corresponding to at least one first image can be obtained, thereby determining the second image with the highest quality score among the at least one first image. This avoids the recognition result being affected by poor image quality, thus effectively improving the accuracy of vehicle state recognition. By training the second model based on at least one second historical image and the labeled quality score of each second historical image, the quality of different second historical images can be learned. Combining the labeled quality score of each second historical image effectively improves the accuracy of the second model in scoring the image quality. By inputting the second image with the highest quality score into the first model, the accuracy of recognizing the target load state of the vehicle is further improved.

[0110] Based on the above embodiments, Figure 3 Flowchart of the vehicle status identification method provided in the embodiments of this application Figure 3 Prior to step S120 above, the method for identifying the vehicle's state further includes the following steps:

[0111] S4, input multiple first historical images into the YOLOv5s model to obtain the predicted loading state corresponding to each first historical image;

[0112] In the YOLOv5s model, the feature fusion Neck layer includes a cross-stage feature fusion C3 module, and the spatial slice channel reorganization Focus module in the YOLOv5s model is replaced with a 6×6 convolution.

[0113] In this step, a cross-stage feature fusion C3 module is added to the feature fusion Neck layer in the YOLOv5s model, and the spatial slice channel recombination Focus module in the YOLOv5s model is replaced with a 6×6 convolution. After inputting multiple first historical images into the YOLOv5s model for feature extraction and feature fusion, the predicted object state corresponding to each first historical image is obtained.

[0114] In one possible implementation, the spatial slice channel reorganization Focus module in the YOLOv5s model is replaced with a 6×6 convolution. Each first historical image is processed by a 6×6 convolution, which allows for feature extraction over a larger range, resulting in the first historical image features. These first historical image features are then fused through a feature fusion Neck layer with an added cross-stage feature fusion C3 module to obtain the predicted loading state for each first historical image.

[0115] In addition, the feature fusion Neck layer of the C3 module, which is used for cross-stage feature fusion, can divide the first historical image features into two groups for separate processing. The first group is directly passed to the next layer without convolution, preserving the first historical image features. The second group performs nonlinear transformation on the first historical image features through multiple bottleneck convolutional blocks to extract the deep features of the first historical image. The first historical image features of the first group and the deep features of the first historical image of the second group are concatenated along the feature channels to obtain the mixed features of the first historical image.

[0116] In one possible implementation, prior to step S4 above, the vehicle state recognition method further includes: preprocessing multiple first historical images, wherein the preprocessing includes: size conversion, three primary colors processing, and pixel filling.

[0117] For example, size conversion is used to indicate that the size of the first historical image is normalized to a square image of a preset size;

[0118] The three primary color processing is used to indicate that the three primary color channel order of the first historical image be converted to the RGB channel order;

[0119] Pixel fill is used to indicate that when the first historical image is resized to obtain a square image, the aspect ratio of the first historical image is maintained, and the blank areas in the first historical image are filled with grayscale values.

[0120] In one possible implementation, multiple first historical images are preprocessed by processing the three primary colors to adjust the channel order of the multiple first historical images, ensuring that the channel order of the first historical images is RGB, maintaining the aspect ratio of the first historical images, and normalizing the first historical images to square images of a preset size. If there are blank areas, grayscale values ​​are used to fill the blank areas.

[0121] S5, determine the loss value based on the unloaded area indicated by each predicted cargo status and the unloaded area indicated by the labeled cargo status of each first historical image.

[0122] For example, the unfilled area includes bounding box parameters, which include: the coordinates of the bounding box center point (x_center, y_center), the width of the bounding box, and the height of the bounding box.

[0123] In one possible implementation, the loss value is calculated based on the predicted bounding box of the unloaded region indicated by each predicted loading state and the labeled bounding box of the unloaded region indicated by the loading state in each first historical image, and can be expressed as follows:

[0124]

[0125] Where CIoU represents the loss value, IoU represents the crossover ratio, and b p b represents the center point of the predicted bounding box. g ρ represents the center point of the labeled bounding box, ρ represents the Euclidean distance between the center point of the predicted bounding box and the center point of the labeled bounding box, c represents the diagonal length of the minimum bounding rectangle, v represents the aspect ratio consistency coefficient, and α represents the aspect ratio consistency weight.

[0126] Furthermore, if each predicted load status indication of an unloaded area includes multiple predicted bounding boxes, the loss value of every two predicted bounding boxes can be calculated. If the loss value is greater than or equal to the loss threshold, the overlap between the two predicted bounding boxes is high, and one of the predicted bounding boxes can be retained while the duplicate predicted bounding boxes are removed.

[0127] S6. Based on the loss value, adjust the parameters in the YOLOv5s model and repeat S4-S6 until the new loss value converges, then determine the YOLOv5s model as the first model.

[0128] For example, if the loss value does not converge, the parameters in the YOLOv5s model are adjusted, and a new loss value is obtained based on the YOLOv5s model with the adjusted parameters. If the new loss value still does not converge, the parameters in the YOLOv5s model are adjusted again until the new loss value converges, and then the YOLOv5s model is determined as the first model.

[0129] In one possible implementation, when the loss value converges, the higher the overlap between the unloaded region indicated by each predicted load state and the unloaded region indicated by the labeled load state in each first historical image, the higher the accuracy of the predicted load state determined by the YOLOv5s model, and the YOLOv5s model can be determined as the first model.

[0130] The vehicle state recognition method provided in this application first inputs multiple first historical images into a YOLOv5s model to obtain the predicted load state corresponding to each first historical image. The feature fusion Neck layer in the YOLOv5s model includes a cross-stage feature fusion C3 module, and the spatial slice channel reconstruction Focus module in the YOLOv5s model is replaced with a 6×6 convolution. Then, based on the unloaded area indicated by each predicted load state and the unloaded area indicated by the labeled load state in each first historical image, a loss value is determined. Finally, based on the loss value, the parameters in the YOLOv5s model are adjusted, and the above steps are repeated until the new loss value converges, and the YOLOv5s model is determined as the first model. In this embodiment, by introducing the C3 module into the Neck layer of the YOLOv5s model, more efficient feature fusion can be achieved. Furthermore, replacing the spatial slice channel reorganization Focus module in the YOLOv5s model with a 6×6 convolution can better capture local features in the image. Multiple first historical images are input into the YOLOv5s model, which extracts features from these images and performs efficient feature fusion, accurately identifying the predicted load state corresponding to each first historical image. Based on the unloaded area indicated by each predicted load state and the unloaded area indicated by the labeled load state in each first historical image, a loss value is determined. This allows the YOLOv5s model to focus on optimizing the recognition accuracy of the unloaded area. By continuously updating the parameters until the loss value converges, the effectiveness of the YOLOv5s model training process and the robustness of the final first model determined by the YOLOv5s model can be ensured, thereby effectively improving the accuracy of vehicle state recognition based on the first model.

[0131] Based on the above embodiments, the following is a vehicle status identification device provided in the embodiments of this application, which can execute the method provided in the above method embodiments.

[0132] Figure 4This is a schematic diagram of the vehicle status recognition device provided in an embodiment of this application. Figure 4 As shown, the vehicle status identification device 400 includes:

[0133] The acquisition module 410 is used to acquire at least one first image of the vehicle when the vehicle is detected to have entered the detection area.

[0134] The determination module 420 is used to determine the target load state of the vehicle based on at least one first image and a first model, wherein the target load state includes: a first state of being fully loaded or a second state of being partially loaded, and the first model is trained based on multiple first historical images and the labeled load state of each first historical image.

[0135] In one or more embodiments, the determining module 420 is specifically used for:

[0136] At least one first image is input into the second model to obtain the second image with the highest quality score among at least one first image. The second model is trained based on at least one second historical image and the labeled quality score of each second historical image.

[0137] The second image is input into the first model to obtain the target load state of the vehicle.

[0138] In one or more embodiments, before inputting the second image into the first model to obtain the target load state of the vehicle, the determination module 420 is further configured to:

[0139] S1, if the quality score of the second image is less than the preset score, obtain at least one new first image from the detection area;

[0140] S2, input at least one new first image into the second model to obtain a new second image with the highest quality score among the at least one new first image;

[0141] S3. If the quality score of the new second image is greater than or equal to the preset score, then the new second image is used as the second image input to the first model. Otherwise, repeat steps S1-S3 until the quality score of the new second image is greater than or equal to the preset score.

[0142] In one or more embodiments, the determining module 420 is further configured to:

[0143] If the target load status indicates that the vehicle is in a second state where it is not fully loaded, mark the unloaded area indicated by the second state and the corresponding confidence data on the second image.

[0144] In one or more embodiments, before determining the target load state of the vehicle based on at least one first image and the first model, the determining module 420 is further configured to:

[0145] S4, input multiple first historical images into the YOLOv5s model to obtain the predicted object state corresponding to each first historical image. The feature fusion Neck layer in the YOLOv5s model includes the cross-stage feature fusion C3 module, and the spatial slice channel reorganization Focus module in the YOLOv5s model is replaced with a 6×6 convolution.

[0146] S5, determine the loss value based on the unloaded area indicated by each predicted cargo status and the unloaded area indicated by the labeled cargo status of each first historical image;

[0147] S6. Based on the loss value, adjust the parameters in the YOLOv5s model and repeat S4-S6 until the new loss value converges, then determine the YOLOv5s model as the first model.

[0148] In one or more embodiments, before inputting at least one first image into the second model to obtain the second image with the highest quality score among the at least one first image, the determining module 420 is further configured to:

[0149] The residual neural network ResNet-18 is trained based on at least one second historical image and the labeled quality score of each second historical image to obtain the second model.

[0150] In one or more embodiments, before inputting multiple first historical images into the YOLOv5s model to obtain the predicted object state corresponding to each first historical image, the determination module 420 is further configured to:

[0151] Multiple first historical images are preprocessed, including: size conversion, three primary colors processing, and pixel filling.

[0152] Based on the above embodiments, Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 5 As shown, the electronic device 500 includes: a processor 510, a memory 520, and a bus 530;

[0153] The memory 520 is used to store the computer-executed instructions of the processor 510;

[0154] The processor 510 is configured to execute the technical solutions of any of the foregoing method embodiments by executing computer execution instructions.

[0155] Optionally, the memory 520 can be either standalone or integrated with the processor 510.

[0156] Optionally, memory 520 may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0157] Bus 530 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one thick line is used to represent a bus in the accompanying drawings of this application, but this does not imply that there is only one bus or one type of bus.

[0158] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0159] The electronic device is used to execute the technical solution of any of the foregoing method embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.

[0160] This application also provides an engineering vehicle, which is used to execute the technical solution provided in any of the above method embodiments or includes the above electronic device, which is used to execute the technical solution provided in any of the above method embodiments.

[0161] This application also provides a computer-readable storage medium storing computer-executable instructions thereon, which, when executed by a processor, are used to implement the technical solutions provided in any of the above method embodiments.

[0162] This application also provides a computer program product, including a computer program, which includes computer instructions stored in a computer-readable storage medium. When the computer program is executed by a processor, it is used to implement the technical solutions provided in any of the above method embodiments.

[0163] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0164] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0165] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0166] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0167] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0168] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0169] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0170] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the scope of this application is limited only by the appended claims.

Claims

1. A method for identifying vehicle status, characterized in that, include: When a vehicle is detected entering the detection area, at least one first image of the vehicle is acquired; Based on the at least one first image and the first model, the target load state of the vehicle is determined, the target load state including: a first state of being fully loaded or a second state of being partially loaded, the first model being trained based on multiple first historical images and the labeled load state of each first historical image.

2. The method according to claim 1, characterized in that, Determining the target cargo state of the vehicle based on the at least one first image and the first model includes: The first image is input into the second model to obtain the second image with the highest quality score among the first images. The second model is trained based on the first image and the labeled quality score of each second historical image. The second image is input into the first model to obtain the target load state of the vehicle.

3. The method according to claim 2, characterized in that, Before inputting the second image into the first model to obtain the target load state of the vehicle, the method further includes: S1, if the quality score of the second image is less than the preset score, obtain at least one new first image from the detection area; S2, input the new at least one first image into the second model to obtain a new second image with the highest quality score among the new at least one first image; S3. If the quality score of the new second image is greater than or equal to the preset score, then the new second image is used as the second image input to the first model. Otherwise, repeat steps S1-S3 until the quality score of the new second image is greater than or equal to the preset score.

4. The method according to claim 2, characterized in that, The method further includes: If the target load status indicates that the vehicle is in a second state of not being fully loaded, mark the unloaded area indicated by the second state and the corresponding confidence level data on the second image.

5. The method according to any one of claims 1-4, characterized in that, Before determining the target load state of the vehicle based on the at least one first image and the first model, the method further includes: S4, input multiple first historical images into the YOLOv5s model to obtain the predicted object state corresponding to each first historical image. The feature fusion Neck layer in the YOLOv5s model includes a cross-stage feature fusion C3 module, and the spatial slice channel reorganization Focus module is replaced with a 6×6 convolution. S5, determine the loss value based on the unloaded area indicated by each predicted cargo status and the unloaded area indicated by the labeled cargo status of each first historical image; S6. Based on the loss value, adjust the parameters in the YOLOv5s model and repeat S4-S6 until the new loss value converges, and determine the YOLOv5s model as the first model.

6. The method according to any one of claims 2-4, characterized in that, Before inputting the at least one first image into the second model to obtain the second image with the highest quality score among the at least one first image, the method further includes: The second model is obtained by training a residual neural network ResNet-18 based on at least one second historical image and the labeled quality score of each second historical image.

7. The method according to claim 5, characterized in that, Before inputting multiple first historical images into the YOLOv5s model to obtain the predicted object state corresponding to each first historical image, the method further includes: The plurality of first historical images are preprocessed, including: size conversion, three primary colors processing, and pixel filling.

8. A vehicle status identification device, characterized in that, include: The acquisition module is used to acquire at least one first image of the vehicle when it is detected that the vehicle has entered the detection area; The determination module determines the target load state of the vehicle based on the at least one first image and the first model. The target load state includes either a first state of being fully loaded or a second state of being partially loaded. The first model is trained based on multiple first historical images and the labeled load state of each first historical image.

9. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to perform the method as described in any one of claims 1-7.

10. An engineering vehicle, characterized in that, The engineering vehicle is used to perform the method as described in any one of claims 1-7 or includes the electronic equipment as described in claim 9.

Images