Method, equipment and medium for judging cleanliness of construction site vehicles based on artificial intelligence

Abstract

The present invention discloses a method, device and medium for judging the cleanliness of construction site vehicles based on artificial intelligence, and the method is as follows: using multiple cameras to capture images of vehicles to be detected; using a target detector to obtain the vehicle and wheel profiles in the image, and correcting the image according to the wheel profile features; comparing the vehicle profile in each corrected image with a given ideal vehicle profile, selecting the image with the greatest similarity as the optimal viewing angle image; using a deep learning method to segment the head, door and cargo box from the optimal viewing angle image; for each part area image, edge detection, color-based image segmentation, and deep learning classification are used to judge whether it is clean, forming 3 pre-judgment results, and then a voting method is used to judge whether each part is clean; for the three parts of the vehicle, if the number of parts judged to be unclean is greater than or equal to 2, the vehicle is judged to be unclean, otherwise the vehicle is judged to be clean. The present invention has a high accuracy rate in judging the cleanliness of construction site vehicles.

Description

Technical Field

[0001] The present invention belongs to the field of image analysis of smart construction sites, and in particular relates to a method, device and medium for judging the cleanliness of construction site vehicles based on artificial intelligence. Background Art

[0002] With the rapid development of urban construction, various construction sites continue to emerge. For the sake of urban civilization and social security, it is necessary to manage the vehicles entering and leaving the construction sites. The cleanliness of the vehicles is a necessary management aspect. Since many of the vehicles on the construction sites are large vehicles, their driving and stopping are somewhat random, and an adaptive and intelligent method is needed for analysis.

[0003] As for the analysis methods, the current aspects mainly include traditional image analysis methods and image analysis methods based on deep learning. These methods are not robust to some vehicle images with uncertain positions. Summary of the invention

[0004] The present invention provides a method, device and medium for judging the cleanliness of construction site vehicles based on artificial intelligence, which has a high accuracy rate in judging the cleanliness of construction site vehicles. To achieve the above technical purpose, the present invention adopts the following technical solutions:

[0005] A method for judging the cleanliness of construction site vehicles based on artificial intelligence, comprising:

[0006] S1, use N cameras to simultaneously capture the side image of the vehicle whose cleanliness is to be judged when it enters or leaves the construction site, N>2;

[0007] S2, using an object detector with instance segmentation, obtains the vehicle outline and wheel outline in each image obtained in S1, and performs rotation correction on the image according to the wheel outline features;

[0008] S3, comparing the vehicle contour in each rotation-corrected image in S2 with the given ideal vehicle contour, calculating the similarity, and selecting the rotation-corrected image with the largest similarity as the optimal viewing angle image;

[0009] S4, using deep learning methods to segment the optimal view image to obtain regional images of the following three parts of the vehicle: the front of the vehicle, the door, and the cargo box;

[0010] S5, for the regional image of each part of the vehicle obtained in S4, three methods, namely edge detection, color-based image segmentation, and deep learning classifier, are used to predict whether the part is clean;

[0011] S6, for each part of the vehicle, a voting method is used to determine whether the part is clean according to the three prediction results obtained in S5;

[0012] S7, for the three parts of the vehicle, namely, the front part, the door and the cargo box, if the number of parts judged to be unclean in S6 is greater than or equal to 2, the vehicle is judged to be unclean, otherwise the vehicle is judged to be clean.

[0013] Furthermore, the method S2 performs rotation correction on the image according to the wheel profile feature is:

[0014] Select the two wheel contours that are farthest apart in the image, calculate the center points of the two wheel contours, connect the two center points into a line, and then rotate the image according to the angle between the connecting line and the bottom boundary of the image so that the connecting line is parallel to the bottom boundary of the image canvas; wherein, before the image is rotated and corrected, the image boundary is parallel to the image screen boundary.

[0015] Furthermore, S3 uses Hausdorff distance to calculate the similarity between two edges.

[0016] Furthermore, the method in which S5 uses edge detection method to pre-judge whether each part of the vehicle is clean is as follows:

[0017] S511, for the region image of the part to be predicted of the vehicle whose cleanliness is to be determined, the canny algorithm is used to extract the part edge in the region image to obtain an edge mask image, in which the pixel value at the edge is 1 and the pixel value at the non-edge is 0;

[0018] S512, performing connected domain analysis on the edges in the edge mask image, and deleting connected domains whose areas are smaller than a preset threshold;

[0019] S513, calculating the sum of the edge pixel values ​​in the edge mask image to obtain the number N of edge pixel points edg ;

[0020] S514, calculate N edge The edge pixel value N of the given vehicle edge *The ratio p=N edg / N edge *, if the ratio p is greater than the preset ratio, the pre-judgment result of the pre-judgment part of the vehicle whose cleanliness is to be judged by the edge detection method is "unclean", otherwise it is "clean".

[0021] Furthermore, S5 uses a color-based image segmentation method to pre-judge whether each part of the vehicle is clean:

[0022] S521, using Gaussian smoothing to perform denoising on the regional image of the part to be predicted of the vehicle whose cleanliness is to be determined;

[0023] S522, using kmeans clustering algorithm, taking RGB channel values ​​as feature values ​​and Euclidean distance as feature distance, clustering and segmenting the denoised regional image in S521, and dividing the denoised regional image into 5 color segmentation regions;

[0024] S523, calculating the average value of each RGB channel of each color segmentation area in S522, and calculating the Euclidean distance of the average value of each RGB channel of the corresponding color segmentation area relative to the given vehicle, and obtaining one Euclidean distance value for each color segmentation area;

[0025] S524, if the maximum value of the five Euclidean distance values ​​obtained in 523 is greater than a preset threshold, the pre-judgment result of the pre-judgment part of the vehicle whose cleanliness is to be judged is "unclean" through the color-based image segmentation method, otherwise it is "clean".

[0026] Furthermore, S5 uses the same deep learning classifier to predict whether each part of the vehicle is clean; the deep learning classifier used is specifically a resnet binary classifier, and its training samples cover clean samples and unclean samples of various parts of the vehicle.

[0027] An electronic device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor implements the method for determining the cleanliness of a construction site vehicle based on artificial intelligence as described in any of the above technical solutions.

[0028] A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for determining the cleanliness of a construction site vehicle based on artificial intelligence as described in any of the above technical solutions is implemented.

[0029] Beneficial Effects

[0030] 1. The present invention selects the image with the best viewing angle with the highest similarity for subsequent processing, which can achieve better recognition effect and is more robust to the vehicle's running position and angle; image segmentation is used to obtain the three parts of the vehicle, namely the front, door and cargo box, so that the characteristics of dirt coverage are more prominent; edge analysis, color clustering analysis and classifier judgment are used for the three parts of the vehicle, namely the front, door and cargo box, which can achieve better recognition effect in the case of small samples and also has better interpretability; finally, the voting method is used to judge the cleanliness of the vehicle parts and the vehicle as a whole, which reduces misjudgment and improves the accuracy of the overall judgment.

[0031] 2. The intermediate results obtained by the present invention from image processing, such as the image with the maximum similarity as the optimal image, can be used to analyze the operation and characteristics of the vehicle, thereby improving the control of the vehicle; such as the cleanliness of each part, can guide managers to better maintain and manage the vehicle.

[0032] 3. The present invention improves the accuracy of vehicle cleanliness recognition and improves the work efficiency and management level of construction sites and construction vehicles through various comprehensive methods including the combination of traditional image processing and deep learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is an overall flow chart of the method described in the embodiment of the present application;

[0034] Figure 2 is a flow chart of the calibration steps described in the embodiment of the present application;

[0035] Figure 3 This is a flowchart of the present application for finding the optimal viewing angle image based on similarity. DETAILED DESCRIPTION

[0036] The following is a detailed description of an embodiment of the present invention. This embodiment is based on the technical solution of the present invention, and provides a detailed implementation method and a specific operation process to further explain the technical solution of the present invention.

[0037] Example 1

[0038] This embodiment provides a method for judging the cleanliness of construction site vehicles based on artificial intelligence. Figure 1 As shown, the following steps are included:

[0039] S1, using N cameras to simultaneously capture the side image of the vehicle whose cleanliness is to be judged when it enters or leaves the construction site. In this embodiment, N=3.

[0040] S2, using an object detector with instance segmentation, obtains the vehicle outline and wheel outline in each image obtained in S1, and performs rotation correction on the image according to the wheel outline features;

[0041] refer to Figure 2 As shown, the rotation correction method for the image in this embodiment is: select the two wheel contours that are farthest apart in the image, calculate the center points of the two wheel contours, connect the two center points into a line, and then rotate the image according to the angle between the connecting line and the bottom boundary of the image, so that the connecting line is parallel to the bottom boundary of the image canvas; wherein, before the image is rotated and corrected, the image boundary is parallel to the image screen boundary.

[0042] Since the subsequent step S3 needs to calculate the similarity of the vehicle contours, it is hoped that the contours of several vehicles can maintain the same reference position first. The actual collected image may not be the reference position due to reasons such as the shooting angle and the vehicle's driving angle. Therefore, the image is corrected through this step to maintain the same reference position and improve the accuracy of the similarity calculation.

[0043] S3, compare the vehicle contour in each rotation-corrected image in S2 with the given ideal vehicle contour, calculate the similarity, select the rotation-corrected image with the largest similarity, and continue to step S4 as the optimal perspective image, such as Figure 3 As shown; specifically, the Hausdorff distance can be used to calculate the similarity between two edges.

[0044] It is difficult for large vehicles on the construction site to stop and reverse, and it is not necessarily possible to drive or stop at a fixed angle or position. Therefore, three cameras are set up at the entrance and exit of the construction site, and an image captured at the most appropriate angle is selected for analysis. Here, the "maximum similarity" indicator is used to find the image captured at the most appropriate perspective.

[0045] S4, using deep learning methods to segment the optimal perspective image, to obtain regional images of the following three parts of the vehicle: the front, the door, and the cargo box.

[0046] The vehicle door is within the range of the vehicle front. Here, both the vehicle door and the vehicle front are analyzed because generally the dirt is on the vehicle door. Therefore, this embodiment needs to analyze the vehicle door in addition to improve the accuracy of the cleaning detection.

[0047] S5, for the regional image of each part of the vehicle obtained in S4, three methods, namely edge detection, color-based image segmentation, and deep learning classifier, are used to predict whether the part is clean.

[0048] Among them, the method of using edge detection method to pre-judge whether each part of the vehicle is clean is:

[0049] S511, for the region image of the part to be predicted of the vehicle whose cleanliness is to be determined, the canny algorithm is used to extract the part edge in the region image to obtain an edge mask image, in which the pixel value at the edge is 1 and the pixel value at the non-edge is 0;

[0050] S512, performing connected domain analysis on the edges in the edge mask image, and deleting connected domains whose areas are smaller than a preset threshold;

[0051] S513, calculating the sum of the edge pixel values ​​in the edge mask image to obtain the number N of edge pixel points edg ;

[0052] S514, calculate Nedge The edge pixel value N of the given vehicle edge *The ratio p=N edg / N edge *, if the ratio p is greater than the preset ratio, the pre-judgment result of the pre-judgment part of the vehicle whose cleanliness is to be judged by the edge detection method is "unclean", otherwise it is "clean".

[0053] For clean vehicles, the edges of these parts are relatively few and relatively fixed. For vehicles covered with dirt, edges will be formed at the dirt, resulting in more edges for unclean vehicles than for clean vehicles. Therefore, this embodiment can pre-judge the vehicle parts based on the ratio of the number of pixels obtained by edge detection to the number of clean vehicles.

[0054] The method of using color-based image segmentation to pre-judge whether each part of the vehicle is clean is:

[0055] S521, using Gaussian smoothing to perform denoising on the regional image of the part to be predicted of the vehicle whose cleanliness is to be determined;

[0056] S522, using kmeans clustering algorithm, taking RGB channel values ​​as feature values ​​and Euclidean distance as feature distance, clustering and segmenting the denoised regional image in S521, and dividing the denoised regional image into 5 color segmentation regions;

[0057] S523, calculating the average value of each RGB channel of each color segmentation area in S522, and calculating the Euclidean distance of the average value of each RGB channel of the corresponding color segmentation area relative to the given vehicle, and obtaining one Euclidean distance value for each color segmentation area;

[0058] S524, if the maximum value of the five Euclidean distance values ​​obtained in 523 is greater than a preset threshold, the pre-judgment result of the pre-judgment part of the vehicle whose cleanliness is to be judged is "unclean" through the color-based image segmentation method, otherwise it is "clean".

[0059] The color of the dirt is different from the color of the vehicle itself. Some parts of the vehicle may have dirt and some may not. Through color clustering image segmentation, the dirt and the original parts can be distinguished. Therefore, this embodiment can pre-judge the vehicle parts based on the distance of the color segmentation area relative to the clean vehicle.

[0060] When using a deep learning classifier to pre-judge whether each part of the vehicle is clean, the same deep learning classifier is used to pre-judge the cleanliness of each part of the vehicle; and the deep learning classifier used is specifically a resnet binary classifier, and the training samples cover clean samples and unclean samples of each part of the vehicle. In addition, for unclean samples, the number of samples of the three parts can be balanced by rotation, perspective transformation, cropping, and color transformation to improve the accuracy of the classifier.

[0061] S6, for each part of the vehicle, a voting method is used to determine whether the part is clean according to the three predicted results obtained in S5.

[0062] S7, for the three parts of the vehicle, namely, the front part, the door and the cargo box, if the number of parts judged to be unclean in S6 is greater than or equal to 2, the vehicle is judged to be unclean, otherwise the vehicle is judged to be clean.

[0063] The present invention has 3 deep learning models:

[0064] [a] Mask-rcnn object detector with instance segmentation is used to detect vehicles, wheels and their contours. In fact, there is an implicit situation here: there are many vehicles on the construction site. Using the object detector will get several vehicles, and the vehicle with the largest detection box is selected as the vehicle to be inspected. Therefore, this step uses the object detector instead of the image segmentation model;

[0065] [b] Image semantic segmentation model, which is used to segment vehicle parts from the vehicle image selected in [a], such as the front of the vehicle, the door, the cargo box, and the outline of the entire vehicle. Since it has been determined to be a vehicle at this step, the use of a semantic model will not cause confusion between different vehicles. The specific model structure that can be used can be unet, deeplab, etc.

[0066] [c] The binary ResNet classification model is used to determine whether the three areas in [b] are clean.

[0067] Example 2

[0068] This embodiment provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor implements the method described in Embodiment 1.

[0069] Example 3

[0070] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method described in Embodiment 1 is implemented.

[0071] The above embodiments are preferred embodiments of the present application. Ordinary technicians in this field can also make various changes or improvements on this basis. Without departing from the overall concept of the present application, these changes or improvements should fall within the scope of protection required by the present application.

Claims

1. A method for judging the cleanliness of construction site vehicles based on artificial intelligence, It is characterized in that include: S1, use N cameras to simultaneously capture the side image of the vehicle whose cleanliness is to be judged when it enters or leaves the construction site, N>2; S2, using an object detector with instance segmentation, obtains the vehicle outline and wheel outline in each image obtained in S1, and performs rotation correction on the image according to the wheel outline features; S3, comparing the vehicle contour in each rotation-corrected image in S2 with the given ideal vehicle contour, calculating the similarity, and selecting the rotation-corrected image with the largest similarity as the optimal viewing angle image; S4, using deep learning methods to segment the optimal view image to obtain regional images of the following three parts of the vehicle: the front of the vehicle, the door, and the cargo box; S5, for the regional image of each part of the vehicle obtained in S4, three methods, namely edge detection, color-based image segmentation, and deep learning classifier, are used to predict whether the part is clean; The method used by S5 to pre-judge whether each part of the vehicle is clean is as follows: S511, for the region image of the part to be predicted of the vehicle whose cleanliness is to be determined, the canny algorithm is used to extract the part edge in the region image to obtain an edge mask image, in which the pixel value at the edge is 1 and the pixel value at the non-edge is 0; S512, performing connected domain analysis on the edges in the edge mask image, and deleting connected domains whose areas are smaller than a preset threshold; S513, calculating the sum of the edge pixel values ​​in the edge mask image to obtain the number N of edge pixel points edg ; S514, calculate N edge The edge pixel value N of the given vehicle edge *The ratio p=N edg / N edge *, if the ratio p is greater than the preset ratio, the pre-judgment result of the pre-judgment part of the vehicle whose cleanliness is to be judged by the edge detection method is "unclean", otherwise it is "clean"; S5 uses a color-based image segmentation method to pre-judge whether each part of the vehicle is clean: S521, using Gaussian smoothing to perform denoising on the regional image of the part to be predicted of the vehicle whose cleanliness is to be determined; S522, using kmeans clustering algorithm, taking RGB channel values ​​as feature values ​​and Euclidean distance as feature distance, clustering and segmenting the denoised regional image in S521, and dividing the denoised regional image into 5 color segmentation regions; S523, calculating the average value of each RGB channel of each color segmentation area in S522, and calculating the Euclidean distance of the average value of each RGB channel of the corresponding color segmentation area relative to the given vehicle, and obtaining one Euclidean distance value for each color segmentation area; S524, if the maximum value of the five Euclidean distance values ​​obtained in S523 is greater than a preset threshold, the pre-judgment result of the pre-judgment part of the vehicle whose cleanliness is to be judged is "unclean" by using a color-based image segmentation method, otherwise it is "clean"; S6, for each part of the vehicle, a voting method is used to determine whether the part is clean according to the three prediction results obtained in S5; S7, for the three parts of the vehicle, namely, the front part, the door and the cargo box, if the number of parts judged to be unclean in S6 is greater than or equal to 2, the vehicle is judged to be unclean, otherwise the vehicle is judged to be clean.

2. The method according to claim 1, It is characterized in that The method of S2 to perform rotation correction on the image according to the wheel profile features is: Select the two wheel contours that are farthest apart in the image, calculate the center points of the two wheel contours, connect the two center points into a line, and then rotate the image according to the angle between the connecting line and the bottom boundary of the image so that the connecting line is parallel to the bottom boundary of the image canvas; wherein, before the image is rotated and corrected, the image boundary is parallel to the image screen boundary.

3. The method according to claim 1, It is characterized in that S3 uses Hausdorff distance to calculate the similarity between two edges.

4. The method according to claim 1, It is characterized in that S5 uses the same deep learning classifier to predict whether each part of the vehicle is clean. The deep learning classifier used is specifically a resnet binary classifier, and its training samples cover clean and unclean samples of various parts of the vehicle.

5. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, It is characterized in that When the computer program is executed by the processor, the processor implements the method according to any one of claims 1 to 4.

6. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images