Muck truck cleanliness calculation method, program product, electronic equipment and storage medium

Abstract

The invention provides a slag car cleanliness calculation method, a program product, electronic equipment and a storage medium. The method comprises the following steps: acquiring a plurality of to-be-detected images with a sequential relationship corresponding to a slag car; using a target tracking algorithm to determine the motion trail of the slag car; according to the motion trail and the timestamp of the to-be-detected image, the washing duration of the muck truck is determined; under the condition that the washing duration meets the requirement, semantic segmentation is carried out on the to-be-detected image, and a sediment area and a slag car contour area are obtained; and determining the cleanliness of the muck truck based on the sediment area and the contour area of the muck truck. According to the motion trails obtained in the to-be-detected images with the time sequence relation, more information is obtained, and the flushing duration of the slag car is calculated more accurately. Whether the washing duration meets the requirement or not is judged firstly, and then semantic segmentation is performed on the to-be-detected image, so that the calculation amount is reduced, the detection efficiency is improved, and the subjective influence caused by manual cleanliness determination is reduced.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to a method for calculating the cleanliness of a muck truck, a program product, an electronic device and a storage medium. Background Art

[0002] Muck trucks are mainly used to transport earth, sand, lime, asphalt, cement and other building materials and construction waste during construction. During or after transportation, muck trucks may carry mud and sand. If muck trucks are on the road with mud, it will not only cause urban road and environmental pollution, but also pose a threat to road operation safety. Therefore, the cleanliness management of muck trucks is very important. The current method usually relies on manual calculation of the cleanliness of muck trucks based on photos of muck trucks and preset cleanliness indicators. This method has high labor costs and low efficiency. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a method, device, electronic device and storage medium for calculating the cleanliness of a muck truck, which are used to reduce the labor cost in the process of calculating the cleanliness of the muck truck and improve efficiency.

[0004] In a first aspect, an embodiment of the present application provides a method for calculating the cleanliness of a dump truck, comprising: obtaining a plurality of images to be tested corresponding to the dump truck and having a time-series relationship; determining a motion trajectory of the dump truck based on the images to be tested; determining a flushing time of the dump truck based on the motion trajectory and a timestamp of the images to be tested; when the flushing time meets the requirements, performing semantic segmentation on the images to be tested to obtain a mud and sand area and a dump truck contour area; and determining the cleanliness of the dump truck based on the mud and sand area and the dump truck contour area.

[0005] In the above implementation process, the motion trajectory is obtained from the images to be tested with a time-series relationship, which obtains more information than analyzing a single image one by one, and improves the accuracy of the subsequent determination of the flushing time of the muck truck. In addition, by first determining whether the flushing time meets the requirements and then performing semantic segmentation on the images to be tested, it is not necessary to calculate the cleanliness of each muck truck, which reduces the amount of calculation and improves the detection efficiency; and through semantic segmentation, while reducing labor costs, it reduces the subjective influence of manually determining the cleanliness of the muck truck.

[0006] Optionally, in an embodiment of the present application, a plurality of images to be tested having a time-series relationship are video frames in a video to be tested corresponding to a dump truck; determining the motion trajectory of the dump truck based on the images to be tested includes: using a target detection algorithm to detect the dump truck in the video frame to obtain a dump truck detection frame; extracting vehicle feature points from the dump truck detection frame; the vehicle feature points include vehicle position information; and obtaining the motion trajectory of the dump truck based on the position changes of the vehicle feature points corresponding to the dump truck in consecutive video frames.

[0007] In the above implementation process, the target detection algorithm and the target tracking algorithm are used for the video frames in the video to be detected to obtain the motion trajectory of the dump truck, so as to realize dynamic perception of the monitoring area; the timing relationship and information complementarity between video frames with timing relationship are utilized to improve the accuracy of target detection, and provide a good data analysis basis for the subsequent flushing time and cleanliness calculation, thereby improving the accuracy of cleanliness.

[0008] Optionally, in an embodiment of the present application, the dump truck includes a target dump truck; according to the position changes of the vehicle feature points corresponding to the dump truck in consecutive video frames, the motion trajectory of the dump truck is obtained, including: if there are multiple dump trucks in the video frame, the vehicle feature points of the target dump truck in the current video frame are matched with multiple vehicle feature points in a video frame after the current video frame, and the successfully matched vehicle feature points are used as the vehicle feature points of the target dump truck in the subsequent video frame; the vehicle feature points also include vehicle visual information; the above steps are repeated with the subsequent video frame as the current video frame to obtain the vehicle feature points of the target dump truck in each video frame; according to the position changes of the vehicle feature points of the target dump truck in consecutive video frames, the motion trajectory of the target dump truck is obtained.

[0009] In the above implementation process, in the case where there are multiple muck trucks in the video frame, the vehicle feature points are matched for the continuous video frames to obtain the vehicle feature points of the target muck truck, and then the motion trajectory of the target muck truck is obtained, so that the motion trajectories of different muck trucks can be distinguished. The flushing time of multiple similar vehicles is superimposed, or the flushing time of the same vehicle is split, so as to improve the accuracy of the calculation of the cleanliness of the muck truck.

[0010] Optionally, in an embodiment of the present application, the flushing duration of the dump truck is determined according to the motion trajectory and the timestamp of the image to be tested, including: obtaining a preset flushing area; determining the start flushing time when the dump truck arrives at the flushing area and the end flushing time when the dump truck leaves the flushing area according to the motion trajectory and the timestamp of the image to be tested; and determining the flushing duration of the dump truck according to the start flushing time and the end flushing time.

[0011] In the above implementation process, the start flushing time when the dump truck arrives at the flushing area and the end flushing time when the dump truck leaves the flushing area are determined according to the motion trajectory and the timestamp of the image to be tested, and then the flushing time of the dump truck is determined. The calculation method is more convenient and the flushing time can be obtained quickly.

[0012] Optionally, in an embodiment of the present application, when the flushing time meets the requirements, semantic segmentation is performed on the image to be tested to obtain the mud and sand area and the outline area of ​​the dump truck, including: when the flushing time meets the requirements, according to the timestamp of the image to be tested, obtaining a cleanliness assessment image after the end of the flushing moment; the cleanliness assessment image includes an image of at least one viewing angle of the dump truck; obtaining a vehicle detection frame corresponding to the dump truck from the cleanliness assessment image; and semantic segmentation is performed on the vehicle detection frame to obtain the mud and sand area and the outline area of ​​the dump truck.

[0013] In the above implementation process, the cleanliness assessment image includes an image of the muck truck from at least one viewing angle, so that semantic segmentation can be performed from multiple angles of the muck truck, the mud and sand area and the muck truck contour area can be determined, and the accuracy of the cleanliness assessment can be improved.

[0014] Optionally, in an embodiment of the present application, if the cleanliness assessment image includes images of a dump truck from multiple viewing angles; performing semantic segmentation on the vehicle detection frame to obtain the mud and sand area and the dump truck contour area, including: performing semantic segmentation on each vehicle detection frame respectively to obtain the mud and sand sub-area and the dump truck contour sub-area corresponding to each vehicle detection frame; obtaining the weight of the vehicle detection frame; and determining the mud and sand area and the dump truck contour area based on the mud and sand sub-area, the dump truck contour sub-area and the weight corresponding to each vehicle detection frame.

[0015] In the above implementation process, when obtaining the mud and sand area and the dump truck contour area, images from multiple perspectives can be considered, and corresponding weights can be configured for the vehicle detection frame. In this way, the calculated mud and sand area and the dump truck contour area are more accurate, and the inaccurate cleanliness calculation will not be caused by inaccurate recognition of a single image, thereby improving the reliability of the cleanliness calculation.

[0016] Optionally, in an embodiment of the present application, the mud and sand area includes a wheel mud and sand area and a body mud and sand area; the dump truck contour area includes a wheel contour and a body contour; based on the mud and sand area and the dump truck contour area, determining the cleanliness of the dump truck, including: calculating the ratio of the wheel mud and sand area to the wheel contour to obtain a first ratio; calculating the ratio of the body mud and sand area to the body contour to obtain a second ratio; obtaining preset wheel weights and body weights, and based on the wheel weights and the body weights, performing weighted summation of the first ratio and the second ratio to obtain the dump truck dirtiness data; determining the cleanliness of the dump truck based on the dump truck dirtiness data.

[0017] In the above implementation process, the wheels and the body are considered in the calculation of the sub-areas, and the ratio of the wheel mud area to the wheel contour and the ratio of the body mud area to the body contour are obtained, so that the calculation of the cleanliness of the muck truck is more accurate. In addition, in the process of calculating the cleanliness of the muck truck, the wheel weight and the body weight are introduced, so that the dirtiness data of the muck truck can be accurately calculated based on the contribution data of the mud in the wheel and the mud in the body to the dirtiness data of the muck truck. The dirtiness data of the muck truck accurately describes the dirtiness of the muck truck, thereby making the cleanliness of the muck truck more accurate and well-founded.

[0018] In the second aspect, an embodiment of the present application also provides a device for calculating the cleanliness of a muck truck, including: an image acquisition module, used to acquire multiple test images with a time series relationship corresponding to the muck truck; a trajectory calculation module, used to determine the motion trajectory of the muck truck based on the test image; a flushing time module, used to determine the flushing time of the muck truck based on the motion trajectory and the timestamp of the test image; a semantic segmentation module, used to perform semantic segmentation on the test image when the flushing time meets the requirements, and obtain the mud and sand area and the muck truck contour area; a cleanliness calculation module, used to determine the cleanliness of the muck truck based on the mud and sand area and the muck truck contour area.

[0019] In a third aspect, an embodiment of the present application further provides a computer program product, including computer program instructions, which, when executed by a processor, execute the method provided by the first aspect or any one of the implementations of the first aspect.

[0020] In a fourth aspect, an embodiment of the present application further provides an electronic device, comprising: a processor and a memory, the memory storing computer program instructions, and the computer program instructions, when executed by the processor, execute the method provided by the first aspect or any one of the implementations of the first aspect.

[0021] In a fifth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method provided by the first aspect or any one of the implementations of the first aspect is executed.

[0022] By adopting a method, program product, electronic device and storage medium for calculating the cleanliness of a muck truck provided by the present application, the motion trajectory is obtained from the image to be tested with a time-series relationship, more information is obtained compared to analyzing a single image one by one, and the accuracy of the subsequent determination of the flushing time of the muck truck is improved. In addition, by first determining whether the flushing time meets the requirements and then performing semantic segmentation on the image to be tested, it is not necessary to calculate the cleanliness of each muck truck, which reduces the amount of calculation and improves the detection efficiency; and through semantic segmentation, while reducing labor costs, it reduces the subjective influence of labor on determining the cleanliness of the muck truck. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 A schematic diagram of a flow chart of a method for calculating the cleanliness of a muck truck provided in an embodiment of the present application;

[0025] Figure 2 A schematic diagram of the structure of a device for calculating the cleanliness of a muck truck provided in an embodiment of the present application;

[0026] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following embodiments of the technical solution of the present application are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present application, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present application.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0029] In the description of the embodiments of the present application, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0030] See also Figure 1 A flow chart of a method for calculating the cleanliness of a muck truck provided in an embodiment of the present application is shown. The method for calculating the cleanliness of a muck truck provided in an embodiment of the present application can be applied to electronic devices, which may include physical devices such as servers, PCs, tablet computers, or smart phones, or virtual devices such as virtual machines or containers. The electronic device may be a single device, or a combination of multiple devices or a cluster of a large number of devices. The method for calculating the cleanliness of a muck truck may include:

[0031] Step S110: Acquire a plurality of images to be tested having a time sequence relationship corresponding to the muck truck.

[0032] Step S120: determining the motion trajectory of the muck truck according to the image to be tested.

[0033] Step S130: Determine the flushing time of the muck truck according to the motion trajectory and the timestamp of the image to be tested.

[0034] Step S140: When the flushing time meets the requirements, semantic segmentation is performed on the image to be tested to obtain the mud and sand area and the muck truck contour area.

[0035] Step S150: Determine the cleanliness of the muck truck based on the mud and sand area and the muck truck contour area.

[0036] In step S110, image acquisition equipment may be deployed in advance in areas where the cleanliness of the muck truck needs to be inspected, such as at the muck truck washing site, parking lot entrance and exit, etc. The camera of the image acquisition equipment is used to collect multiple images to be tested corresponding to the muck truck with a time sequence relationship.

[0037] The multiple images to be tested with a time-series relationship may be video frames in the video to be tested corresponding to the muck truck, wherein the video frames may be continuous or at preset intervals, for example, one frame every 5 seconds, or one frame every 5 frames, etc., which is not limited in this application. The multiple images to be tested with a time-series relationship may also be multiple photos taken by a camera at preset intervals, for example, one image may be taken every 10 seconds, and the pictures within the preset time constitute the images to be tested with a time-series relationship.

[0038] In step S120, the motion trajectory of the muck truck is determined based on a plurality of images to be tested that have a time-series relationship. The images to be tested may be preprocessed first, for example, the collected images to be tested may be denoised to eliminate noise interference in the images to be tested; the images to be tested may also be enhanced to improve the contrast and clarity of the images to be tested, so as to facilitate subsequent target detection.

[0039] After that, a suitable target detection algorithm can be used to identify the muck trucks in the image to be tested, and each muck truck can be used as a target for motion trajectory tracking. Of course, the target muck truck in the image to be tested can also be used as a target, and only the motion trajectory of the target muck truck can be tracked. The motion trajectory tracking can use a continuous detection or tracking algorithm to record the position information of the muck truck obtained during the tracking process as a trajectory point, and form a complete motion trajectory of the muck truck based on the trajectory points. This process will be described in detail later.

[0040] In step S130, the timestamp of the image to be tested can be obtained by using the built-in function of recording timestamps in the image acquisition device, or by using SDK (Software Development Kit) or API (Interface). If the image acquisition device itself does not support direct recording of timestamps, the shooting time of the image to be tested can be obtained as the timestamp by using the date and time display in the image to be tested.

[0041] The motion trajectory records information such as the driving direction and / or driving path of the muck truck. Combined with the timestamp of the image to be tested, the flushing time used by the muck truck during the flushing process can be determined.

[0042] In step S140, considering that the calculation of cleanliness of all muck trucks may consume a large amount of computing resources, before the calculation of cleanliness, after obtaining the flushing water duration, it is possible to first determine whether the flushing meets the requirements. For example, whether the flushing duration reaches a threshold, different types of muck trucks can preset corresponding thresholds, and muck trucks transporting different materials can set corresponding thresholds respectively.

[0043] When the flushing time meets the requirements, the image to be tested is semantically segmented, which can be achieved through machine learning algorithms. For example, the collected training images are annotated at the pixel level using image annotation tools to clearly distinguish the mud and sand areas, the outline of the dump truck, and other possible background or interference objects. Select a suitable model and train the model using annotated data sets (multiple training images). During the training process, by adjusting the parameters of the model (such as learning rate, batch size, etc.) and optimization algorithms (such as gradient descent, Adam, etc.), the model gradually learns how to accurately classify the pixels in the image into mud and sand areas, dump truck outlines, or other categories. Then, the trained model is used to perform semantic segmentation on the new test image to obtain the mud and sand areas and the dump truck outline areas in the image to be tested. Since semantic segmentation can be pixel-level, after segmentation, the pixel areas corresponding to the mud and sand areas and the dump truck outline areas can be obtained.

[0044] In an optional embodiment, before semantic segmentation is performed on the image to be tested, target detection may be performed on the image to be tested to determine a vehicle detection frame in the image to be tested. Thereafter, in the process of semantic recognition, only the image corresponding to the vehicle detection frame may be recognized, thereby effectively eliminating the interference of the background area in the image to be tested on the semantic recognition and improving the recognition accuracy and efficiency.

[0045] In step S150, the mud and sand area and the muck truck contour area are obtained through semantic segmentation. The larger the proportion of the mud and sand area in the muck truck contour, the lower the cleanliness. Alternatively, weights are assigned to the mud and sand area and the muck truck contour area respectively, and the cleanliness of the muck truck is determined based on the mud and sand area, the muck truck contour area, and the weights corresponding to the two. This method will be introduced later.

[0046] In the implementation process of the above embodiment, the motion trajectory is obtained from the image to be tested with a time-series relationship, and more information is obtained compared to analyzing a single image one by one, thereby improving the accuracy of the subsequent determination of the flushing time of the muck truck. And by first determining whether the flushing time meets the requirements, and then performing semantic segmentation on the image to be tested, the amount of calculation is reduced and the detection efficiency is improved. And the semantic segmentation of the detection image is performed to accurately obtain the mud and sand area and the muck truck contour area, and then obtain a more accurate and more unified muck truck cleanliness, reducing the subjective influence of manual determination of cleanliness.

[0047] Optionally, in an embodiment of the present application, the multiple images to be tested having a time-series relationship are video frames in a video to be tested corresponding to the muck truck, and the video to be tested can be acquired by the aforementioned deployed image acquisition device.

[0048] Determining the motion trajectory of the muck truck according to the image to be tested includes the following steps:

[0049] Use the target detection algorithm to detect the muck truck in the video frame and obtain the muck truck detection frame. For example, the pre-trained YOLOv8 model can be used to detect the muck truck in the image to be tested, and then the bounding box (Bounding Boxes, BBoxes) is obtained according to the detection result of the muck truck. The bounding box is the muck truck detection frame. Of course, the target detection algorithm can also use other algorithms such as Faster R-CNN algorithm or other YOLO models, etc., which are not limited in this application.

[0050] Extract vehicle feature points from the muck truck detection frame; the vehicle feature points include vehicle position information. According to the position changes of the vehicle feature points corresponding to the muck truck in consecutive video frames, the movement trajectory of the muck truck is obtained. For example, the detected vehicle information is input into the DeepSORT tracker, which uses the Kalman filter to predict the vehicle position information, uses the deep features for matching, and outputs the updated vehicle ID and trajectory information.

[0051] The position change of vehicle feature points, such as coordinate points, can be smoothed on the trajectory points in consecutive video frames to eliminate the trajectory jitter caused by detection errors or fluctuations in the tracking algorithm. The smoothed trajectory points are connected to form a complete movement trajectory of the muck truck.

[0052] As an implementation method, a tracking algorithm such as a particle filter or a correlation filter can also be used to continuously track the detected muck truck. These algorithms can predict the position of the muck truck in the next frame according to the position change between consecutive frames, thereby tracking its motion trajectory.

[0053] In the implementation process of the above embodiment: the target detection algorithm and the target tracking algorithm are used for the video frames in the video to be detected to obtain the motion trajectory of the dump truck, so as to realize dynamic perception of the monitoring area; the timing relationship and information complementarity between the video frames with timing relationship are utilized to improve the accuracy of target detection, provide a good data analysis basis for the subsequent flushing time and cleanliness calculation, and thus improve the accuracy of cleanliness.

[0054] Optionally, in an embodiment of the present application, considering that there may be multiple dump trucks in the monitoring area at the same time, multiple dump trucks will also appear in the image to be tested. Therefore, when performing trajectory calculation, the motion trajectory corresponding to each two dump truck generators should be used. In this way, the flushing time of each two dump trucks can be accurately calculated. If the dump trucks are not distinguished, the flushing time of multiple similar vehicles may be superimposed, or the flushing time of the same vehicle may be split, which will affect the subsequent calculation of the cleanliness of the dump truck.

[0055] The muck truck includes a target muck truck, which can be any muck truck in the image to be tested, and there can be multiple target muck trucks. For example, a first target muck truck, a second target muck truck, etc. can be further distinguished to achieve simultaneous tracking of motion trajectories of multiple muck trucks. In the first frame, all detected muck trucks are initialized into a tracking list, and a unique ID is assigned to each muck truck.

[0056] According to the position changes of the vehicle feature points corresponding to the muck truck in the continuous video frames, the motion trajectory of the muck truck is obtained, including:

[0057] If there are multiple muck trucks in the video frame, the vehicle feature points of the target muck truck in the current video frame are matched with multiple vehicle feature points in the next video frame of the current video frame, and the successfully matched vehicle feature points are used as the vehicle feature points of the target muck truck in the next video frame; the aforementioned vehicle feature points include vehicle position information, and further vehicle feature points also include vehicle visual information. In the matching process, matching can be performed for vehicle position information and / or vehicle visual information. Among them, the vehicle position information can be the coordinate point, pixel coordinate or vehicle detection frame position of the muck truck, and the vehicle visual information can be the color, height, vehicle type and other information of the vehicle.

[0058] The matching process, for example, calculates the similarity (such as IOU, i.e., intersection over union) of the vehicle feature points in the current video frame and multiple vehicle feature points in the next video frame of the current video frame. Then, a data association algorithm (such as the Hungarian algorithm) is used to find the best match from the multiple vehicle feature points. The best match is the successfully matched vehicle feature point, which is used as the vehicle feature point of the target muck truck in the next video frame.

[0059] If the detected vehicle feature points of the current video frame cannot match multiple vehicle feature points in the video frame after the current video frame, it may be a newly appeared muck truck, which needs to be added to the tracking list and assigned a new ID.

[0060] The above steps are repeated with the next video frame as the current video frame to obtain the vehicle feature points of the target muck truck in each video frame. For example, with the first frame as the current video frame, the vehicle feature points of the target muck truck in the second frame are matched, and then the second frame is used as the current video frame, and the vehicle feature points of the target muck truck in the third frame are matched according to the vehicle feature points of the target muck truck in the second frame, and so on, until the vehicle feature points of the target muck truck in each video frame are obtained.

[0061] According to the position change of the vehicle feature points of the target muck truck in the continuous video frames, the motion trajectory of the target muck truck is obtained. After the vehicle feature points of the target muck truck in each video frame are obtained, the other vehicle feature points in the video frame will not affect the formation of the motion trajectory of the target muck truck, and the motion trajectories of different muck trucks can be distinguished.

[0062] In the implementation process of the above embodiment: in the case where there are multiple muck trucks in the video frame, the vehicle feature points are matched for the continuous video frames to obtain the vehicle feature points of the target muck truck, and then the motion trajectory of the target muck truck is obtained, so as to distinguish the motion trajectories of different muck trucks. The flushing time of multiple similar vehicles is superimposed, or the flushing time of the same vehicle is split, so as to improve the accuracy of the calculation of the cleanliness of the muck truck.

[0063] Optionally, in the embodiment of the present application, the flushing time of the muck truck is determined according to the motion trajectory and the timestamp of the image to be measured, including:

[0064] A preset flushing area is obtained. The flushing area can be preset, and the flushing area also has a corresponding area in the image to be tested.

[0065] According to the motion trajectory and the timestamp of the image to be tested, the start flushing time when the muck truck arrives at the flushing area and the end flushing time when the muck truck leaves the flushing area are determined. The motion trajectory can be used to know whether the muck truck has reached the flushing area, obtain the timestamp of the image to be tested when the muck truck arrives at the flushing area, and take the time when the muck truck arrives at the flushing area as the start flushing time. And obtain the timestamp of the image to be tested when the muck truck leaves the flushing area, and take the time when the muck truck leaves the flushing area as the end flushing time.

[0066] The flushing duration of the muck truck is determined according to the flushing start time and the flushing end time, that is, the interval from the flushing start time to the flushing end time is the flushing duration of the muck truck.

[0067] As an implementation method, another method can also be used to determine the flushing time of the muck truck. This method, for example, is as follows: according to the motion trajectory, the number of images to be tested (or the number of video frames) corresponding to the time when the muck truck arrives at the flushing area and leaves the flushing area is determined, and the flushing time of the muck truck is determined according to the number of images to be tested and the preset interval between each image to be tested. This method is suitable for scenarios where the images to be tested captured by the image acquisition device do not carry a timestamp, and this method can still accurately calculate the flushing time of the muck truck.

[0068] In the implementation process of the above embodiment: according to the motion trajectory and the timestamp of the image to be tested, the start flushing time when the dump truck arrives at the flushing area and the end flushing time when the dump truck leaves the flushing area are determined, and then the flushing time of the dump truck is determined. The calculation method is more convenient and the flushing time can be obtained quickly.

[0069] Optionally, in the embodiment of the present application, when the flushing time meets the requirements, semantic segmentation is performed on the image to be tested to obtain the mud and sand area and the muck truck contour area, including:

[0070] When the flushing time meets the requirements, a cleanliness assessment image after the flushing is finished is obtained according to the timestamp of the image to be tested; the cleanliness assessment image includes an image of at least one viewing angle of the muck truck; the viewing angles are such as the front, rear, left, right and top view of the muck truck. The cleanliness assessment image can be taken after the flushing is finished.

[0071] A vehicle detection frame corresponding to the muck truck is obtained from the cleanliness assessment image. For example, a pre-trained YOLOv8 model may be used to detect the muck truck in the cleanliness assessment image, and then the vehicle detection frame is obtained according to the detection result of the muck truck.

[0072] The vehicle detection frame is semantically segmented to obtain the mud and sand area and the muck truck contour area. If there are cleanliness assessment images at multiple viewing angles, the vehicle detection frame corresponding to the cleanliness assessment image at each viewing angle can be obtained. The image corresponding to each vehicle detection frame needs to be semantically segmented. The implementation process of semantic segmentation is as mentioned above.

[0073] In the implementation process of the above embodiment: the cleanliness assessment image includes an image of at least one viewing angle of the muck truck, so that semantic segmentation can be performed from multiple angles of the muck truck, the mud and sand area and the muck truck contour area can be determined, and the accuracy of the cleanliness assessment can be improved.

[0074] Optionally, in an embodiment of the present application, if the cleanliness assessment image includes images of a dump truck from multiple viewing angles; for example, the cleanliness assessment image can be five views of the dump truck from five viewing angles: front, rear, left, right, and top view, and of course there can be more or less.

[0075] Perform semantic segmentation on the vehicle detection frame to obtain the mud and sand area and the muck truck contour area, including:

[0076] Each cleanliness assessment image will be subjected to target detection to obtain the vehicle detection frame corresponding to the muck truck. For example, the five views mentioned above will generate a vehicle detection frame corresponding to the viewing angle for each cleanliness assessment image. Semantic segmentation is performed on each vehicle detection frame to obtain the mud and sand sub-region and muck truck contour sub-region corresponding to each vehicle detection frame. This step refers to the above.

[0077] The weight of the vehicle detection frame is obtained. The weight of the vehicle detection frame may refer to the weight of the entire vehicle detection frame, or the weight of the mud and sand sub-area and the muck truck contour sub-area in the vehicle detection frame.

[0078] The weight can be determined according to the image quality of each cleanliness assessment image. For example, the higher the image quality of the cleanliness assessment image, the higher the weight of the corresponding vehicle detection frame. As an implementation method, the weight of the vehicle detection frame can also be determined based on the confidence data by obtaining the confidence data of the sediment sub-regions and the muck truck contour sub-regions in the vehicle detection frame. The process of obtaining the confidence data can be: the probability that each pixel belongs to a specific category (sediment area and muck truck contour area) is obtained through the softmax layer, and the confidence data of the sediment area and the muck truck contour area are counted. The higher the confidence data, the higher the reliability of the area, and the higher the weight.

[0079] Based on the mud and sand sub-regions, the muck truck contour sub-regions and the weights corresponding to each vehicle detection frame, the mud and sand regions and the muck truck contour regions are determined. For example, the mud and sand sub-regions are weighted and summed to obtain the mud and sand regions; the muck truck contour sub-regions are weighted and summed to obtain the muck truck contour regions. In one embodiment, if the weights of each vehicle detection frame are consistent, the mud and sand sub-regions corresponding to all vehicle detection frames can be summed to obtain the mud and sand regions; the muck truck contour sub-regions corresponding to all vehicle detection frames can be summed to obtain the muck truck contour regions.

[0080] In the implementation process of the above embodiment: when obtaining the mud and sand area and the dump truck outline area, images of multiple viewing angles can be considered, and corresponding weights can be configured for the vehicle detection frame. In this way, the calculated mud and sand area and the dump truck outline area are more accurate, and the inaccurate cleanliness calculation will not be caused by inaccurate recognition of a single image, thereby improving the reliability of the cleanliness calculation.

[0081] Optionally, in the embodiment of the present application, the mud and sand area includes the wheel mud and sand area and the body mud and sand area; the muck truck contour area includes the wheel contour and the body contour; based on the mud and sand area and the muck truck contour area, determining the cleanliness of the muck truck includes:

[0082] Calculating a ratio of a wheel mud and sand area to a wheel profile to obtain a first ratio;

[0083] Calculating the ratio of the mud and sand area of ​​the vehicle body to the contour of the vehicle body to obtain a second ratio;

[0084] The cleanliness of the muck truck is determined according to the first ratio and the second ratio. For example, the first ratio and the second ratio may be added to obtain a total value, and the cleanliness of the corresponding muck truck is determined based on the total value and a preset index. The higher the total value, the larger the area of ​​silt, and the lower the cleanliness of the muck truck.

[0085] In the implementation process of the above embodiment: the wheels and the body are considered in the calculated sub-areas, and the ratio of the wheel mud area to the wheel contour and the ratio of the body mud area to the body contour are obtained, so that the calculation of the cleanliness of the muck truck is more accurate.

[0086] Optionally, in the embodiment of the present application, considering that the mud and sand in the wheels are more likely to cause road pollution than the mud and sand in the body, the embodiment of the present application introduces wheel weights and body weights.

[0087] Determining the cleanliness of the muck truck according to the first ratio and the second ratio comprises the following steps:

[0088] Get the preset wheel weight and body weight. Since the mud and sand in the wheel are more likely to cause road pollution, the wheel weight can be greater than the body weight, and the sum of the wheel weight and the body weight is 1. The wheel weight represents the contribution of the mud and sand in the wheel to the dirtiness data of the muck truck, and the body weight represents the contribution of the mud and sand in the body to the dirtiness data of the muck truck.

[0089] Based on the wheel weight and the body weight, the first ratio and the second ratio are weighted and summed to obtain the dirtiness data of the muck truck. The product of the first ratio and the wheel weight is added to the product of the second ratio and the body weight to obtain the dirtiness data of the muck truck. The value of the dirtiness data of the muck truck can be [0,1].

[0090] The cleanliness of the muck truck is determined according to the dirtiness data of the muck truck. The calculation formula of the cleanliness of the muck truck can be: C = 1-D; where C represents the cleanliness of the muck truck and D represents the dirtiness data of the muck truck.

[0091] In the implementation process of the above embodiment: in the process of calculating the cleanliness of the dump truck, the wheel weight and the body weight are introduced, so that the dirtiness data of the dump truck can be accurately calculated according to the contribution data of the mud and sand in the wheels and the mud and sand in the body to the dirtiness data of the dump truck. The dirtiness data of the dump truck accurately describes the dirtiness of the dump truck, thereby making the cleanliness of the dump truck more accurate and well-founded.

[0092] See also Figure 2 The structural schematic diagram of the device for calculating the cleanliness of a muck truck provided in an embodiment of the present application is shown; the embodiment of the present application provides a device 200 for calculating the cleanliness of a muck truck, comprising:

[0093] An image acquisition module 210 is used to acquire a plurality of images to be tested having a time sequence relationship corresponding to the muck truck;

[0094] A trajectory calculation module 220, used to determine the motion trajectory of the muck truck according to the image to be tested;

[0095] A flushing duration module 230 is used to determine the flushing duration of the muck truck according to the motion trajectory and the timestamp of the image to be tested;

[0096] The semantic segmentation module 240 is used to perform semantic segmentation on the image to be tested to obtain the sediment area and the muck truck contour area when the flushing time meets the requirements;

[0097] The cleanliness calculation module 250 is used to determine the cleanliness of the muck truck based on the mud and sand area and the muck truck contour area.

[0098] Optionally, in an embodiment of the present application, a device 200 for calculating the cleanliness of a dump truck, a plurality of images to be tested having a time-series relationship are video frames in a video to be detected corresponding to the dump truck; a trajectory calculation module 220 is specifically used to detect the dump truck in the video frame using a target detection algorithm, and obtain a dump truck detection frame; extract vehicle feature points from the dump truck detection frame; the vehicle feature points include vehicle position information; and obtain the movement trajectory of the dump truck according to the position changes of the vehicle feature points corresponding to the dump truck in continuous video frames.

[0099] Optionally, in an embodiment of the present application, the dump truck cleanliness calculation device 200 includes a target dump truck; the trajectory calculation module 220 is also used for matching the vehicle feature points of the target dump truck in the current video frame with multiple vehicle feature points in a video frame subsequent to the current video frame if there are multiple dump trucks in the video frame, and using the successfully matched vehicle feature points as the vehicle feature points of the target dump truck in the subsequent video frame; the vehicle feature points also include vehicle visual information; the above steps are repeated with the subsequent video frame as the current video frame to obtain the vehicle feature points of the target dump truck in each video frame; and the motion trajectory of the target dump truck is obtained according to the position changes of the vehicle feature points of the target dump truck in consecutive video frames.

[0100] Optionally, in an embodiment of the present application, the muck truck cleanliness calculation device 200 and the flushing time module 230 are used to obtain a preset flushing area; determine the start flushing time when the muck truck arrives at the flushing area and the end flushing time when the muck truck leaves the flushing area according to the motion trajectory and the timestamp of the image to be measured; and determine the flushing time of the muck truck according to the start flushing time and the end flushing time.

[0101] Optionally, in an embodiment of the present application, the cleanliness calculation device 200 of the dump truck and the semantic segmentation module 240 are specifically used to obtain a cleanliness assessment image after the end of flushing according to the timestamp of the image to be tested when the flushing time meets the requirements; the cleanliness assessment image includes an image of at least one viewing angle of the dump truck; a vehicle detection frame corresponding to the dump truck is obtained from the cleanliness assessment image; and the vehicle detection frame is semantically segmented to obtain a mud and sand area and a dump truck contour area.

[0102] Optionally, in an embodiment of the present application, the dump truck cleanliness calculation device 200, if the cleanliness evaluation image includes images of the dump truck at multiple viewing angles; the semantic segmentation module 240 is also used to perform semantic segmentation on each vehicle detection frame respectively, to obtain the mud and sand sub-region and the dump truck contour sub-region corresponding to each vehicle detection frame; to obtain the weight of the vehicle detection frame; and to determine the mud and sand region and the dump truck contour region based on the mud and sand sub-region, the dump truck contour sub-region and the weight corresponding to each vehicle detection frame.

[0103] Optionally, in an embodiment of the present application, in the device 200 for calculating the cleanliness of a dump truck, the mud and sand area includes the wheel mud and sand area and the body mud and sand area; the contour area of ​​the dump truck includes the wheel contour and the body contour; the cleanliness calculation module 250 is specifically used to calculate the ratio of the wheel mud and sand area to the wheel contour to obtain a first ratio; calculate the ratio of the body mud and sand area to the body contour to obtain a second ratio; and determine the cleanliness of the dump truck based on the first ratio and the second ratio.

[0104] Optionally, in an embodiment of the present application, the muck truck cleanliness calculation device 200 and the cleanliness calculation module 250 are specifically used to obtain preset wheel weights and body weights, and based on the wheel weights and the body weights, perform weighted summation on the first ratio and the second ratio to obtain the dirtiness data of the muck truck; and determine the cleanliness of the muck truck based on the dirtiness data of the muck truck.

[0105] It should be understood that the device corresponds to the above-mentioned embodiment of the method for calculating the cleanliness of a muck truck, and can execute the various steps involved in the above-mentioned method embodiment. The specific functions of the device can be found in the description above. To avoid repetition, the detailed description is appropriately omitted here. The device includes at least one software function module that can be stored in a memory in the form of software or firmware or solidified in the operating system (OS) of the device.

[0106] See also Figure 3 The electronic device 300 provided in the embodiment of the present application includes: a processor 310 and a memory 320, wherein the memory 320 stores machine-readable instructions executable by the processor 310, and when the machine-readable instructions are executed by the processor 310, the above method is executed.

[0107] An embodiment of the present application further provides a storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above method is executed.

[0108] Among them, the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable red-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, disk or optical disk.

[0109] In several embodiments provided by the embodiments of the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and a part of the module, program segment or code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0110] In addition, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part.

[0111] The above description is only an optional implementation manner of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the embodiments of the present application, which should be covered within the protection scope of the embodiments of the present application.

Claims

1. A method for calculating the cleanliness of a muck truck, characterized in that: include: Acquire multiple images to be tested with a time sequence relationship corresponding to the muck truck; Determine the motion trajectory of the muck truck according to the image to be tested; Determine the flushing duration of the muck truck according to the motion trajectory and the timestamp of the image to be tested; When the flushing time meets the requirements, semantic segmentation is performed on the image to be tested to obtain the sediment area and the muck truck contour area; The cleanliness of the muck truck is determined based on the mud and sand area and the muck truck contour area.

2. The method according to claim 1, characterized in that The plurality of images to be tested having a time sequence relationship are video frames in a video to be detected corresponding to the muck truck; and determining the motion trajectory of the muck truck according to the images to be tested includes: Detect the muck truck in the video frame using a target detection algorithm to obtain a muck truck detection frame; Extracting vehicle feature points from the muck truck detection frame; the vehicle feature points include vehicle position information; The motion trajectory of the muck truck is obtained according to the position changes of the vehicle feature points corresponding to the muck truck in continuous video frames.

3. The method according to claim 2, characterized in that The muck truck includes a target muck truck; and obtaining the motion trajectory of the muck truck according to the position change of the vehicle feature points corresponding to the muck truck in continuous video frames includes: If there are multiple muck trucks in the video frame, the vehicle feature points of the target muck truck in the current video frame are matched with multiple vehicle feature points in a video frame after the current video frame, and the successfully matched vehicle feature points are used as the vehicle feature points of the target muck truck in the subsequent video frame; the vehicle feature points also include vehicle visual information; The following video frame is used as the current video frame to repeat the above steps to obtain the vehicle feature points of the target muck truck in each video frame; The motion trajectory of the target muck truck is obtained according to the position change of the vehicle feature points of the target muck truck in continuous video frames.

4. The method according to claim 1, characterized in that: Determining the flushing duration of the muck truck according to the motion trajectory and the timestamp of the image to be measured includes: Get the preset flushing area; Determine, according to the motion trajectory and the timestamp of the image to be measured, the start time of flushing when the muck truck arrives at the flushing area and the end time of flushing when the muck truck leaves the flushing area; The flushing duration of the muck truck is determined according to the flushing start time and the flushing end time.

5. The method according to claim 1, characterized in that When the flushing time meets the requirements, semantic segmentation is performed on the image to be tested to obtain the sediment area and the muck truck contour area, including: When the flushing time meets the requirements, a cleanliness assessment image after the flushing is finished is acquired according to the timestamp of the image to be tested; the cleanliness assessment image includes an image of at least one viewing angle of the muck truck; Acquire a vehicle detection frame corresponding to the muck truck from the cleanliness evaluation image; The vehicle detection frame is semantically segmented to obtain a mud and sand area and a muck truck contour area.

6. The method according to claim 5, characterized in that If the cleanliness assessment image includes images of the muck truck at multiple viewing angles; the semantic segmentation of the vehicle detection frame to obtain the mud and sand area and the muck truck contour area includes: Semantically segmenting each of the vehicle detection frames to obtain the sediment sub-region and the muck truck contour sub-region corresponding to each of the vehicle detection frames; Obtaining a weight of the vehicle detection frame; The mud and sand area and the muck truck contour area are determined based on the mud and sand sub-areas, the muck truck contour sub-areas and the weights corresponding to each of the vehicle detection frames.

7. The method according to claim 1, characterized in that The mud and sand area includes a wheel mud and sand area and a body mud and sand area; the muck truck contour area includes a wheel contour and a body contour; and determining the cleanliness of the muck truck based on the mud and sand area and the muck truck contour area includes: Calculating a ratio of the wheel mud and sand area to the wheel profile to obtain a first ratio; Calculating a ratio of the mud and sand area of ​​the vehicle body to the contour of the vehicle body to obtain a second ratio; Get the preset wheel weight and body weight, Based on the wheel weight and the body weight, weighted summing the first ratio and the second ratio is performed to obtain dirtiness data of the muck truck; The cleanliness of the muck truck is determined according to the dirtiness data of the muck truck.

8. A computer program product, characterized in that The method comprises computer program instructions, and when the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is executed.

9. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the method according to any one of claims 1 to 7 is executed.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is executed.

Images