A missed detection detection method for road disease target detection

Abstract

The present application relates to a kind of missed detection detection methods for road surface disease target detection, belong to highway pavement disease detection and quality control field.The present application proposes a kind of "pavement completion" image processing technology, can be replaced into normal pavement with any bounding box simultaneously, and the data set of missed detection model is made, and missed detection model is trained.Missed detection model can automatically detect whether the area in the picture except bounding box exists larger and obvious disease missed detection problem, and make good preparation for subsequent disease supplement mark work.The method of the present application can automatically improve the disease missed detection situation in the picture of road surface disease target detection result.

Description

Technical Field

[0001] This invention relates to a method for detecting missed defects in road surface defects, belonging to the field of highway pavement defect detection and quality control. Background Technology

[0002] Asphalt pavement is one of the most widely used high-grade pavements in road construction. With the development of China's domestic asphalt material industry, asphalt pavement has been widely applied to urban roads and highways, becoming the high-grade pavement with the largest paved area in China. However, with increasing traffic volume and frequent weather changes, various pavement defects are constantly emerging, including but not limited to cracks, repairs, and potholes. This invention addresses six different pavement defects: cracks, alligator cracking, loosening, potholes, bleeding, and repairs. Currently, the mainstream methods for detecting pavement defects in the computer field are deep learning-based object detection methods. However, the pavement defect object detection images obtained using these methods inevitably have a small number of missed defects. Current solutions to this problem mainly include manual review and model-based review. Manual review can consume a lot of time and manpower when dealing with a large number of object detection images. While model-based review theoretically consumes fewer resources than manual methods, there are currently no mature implementation examples, so its effectiveness is unknown.

[0003] The current feasible method for detecting missed defects relies on manual review of each target detection image. This involves checking for misdetected defects within the bounding box and for any missed defects outside the bounding box. Only after multiple reviews and corrections can a more accurate target detection result image be obtained. However, this method has several shortcomings, including: first, the human and material resources consumed increase dramatically with the number of target detection images; second, review takes a significant amount of time that is difficult to reduce; third, the long hours and repetitive nature of the work can lead to fatigue and decreased concentration among reviewers, resulting in inefficient and substandard review quality; and fourth, inconsistent annotation levels among reviewers, leading to subjectivity and potentially inconsistent results. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide a method for detecting missed detections of pavement defects. This method aims to address the issue that a small number of missed detections exist in the pavement defect detection result map, and that currently only manual review can detect these cases, which results in high resource consumption, high time costs, low efficiency due to the subjectivity of annotation, and inconsistent results.

[0005] The technical solution of this invention is as follows: This invention designs a dataset preprocessing method for image processing technology called "road surface completion". This method can uniformly process the hand-labeled dataset of road surface defects into a dataset for a missed detection model. Using this dataset, a missed detection model (binary classification) can be trained. This model can detect whether there are still large and obvious defects in any road surface completion image. If the detection is a positive sample, there is no missed detection problem; if the detection is a negative sample, there is a missed detection problem, and it is necessary to use a target detection annotation tool to supplement the defect annotation. This invention attempts to fill the gap in the field of using models to review missed detection results, and has the potential for automatic and batch detection of missed detection problems.

[0006] A method for detecting missed detections of pavement defects, comprising the following steps:

[0007] Step 1: Collect manually labeled data sets of road surface defects;

[0008] Step 2: Use the "road surface completion" image processing technology to convert the manually labeled road surface defect dataset into a binary classification dataset;

[0009] Step 3: Train the missed detection model using a binary classification dataset, and perform missed detection to obtain the missed detection results;

[0010] Step 4: Analyze the missed detection results. If there are any missed detections, use the target detection and annotation tool to supplement and mark the missed road surface defects.

[0011] Specifically, Step 1 is as follows:

[0012] Step 1.1: Obtain a dataset of manually labeled road surface defects of six different types and the same number. The images are all grayscale images collected by the inspection vehicle, with an image size of 2048*2048, and are stored in the image subfolders under the six folders respectively.

[0013] Step 1.2: Based on the file names of the manually labeled pavement defects dataset, find the corresponding label files and store them uniformly in the label subfolders under the six folders;

[0014] Step 1.3: Use the object detection and annotation tool to check the annotation status of the manually annotated dataset of road surface defects to ensure that there are no missing annotations.

[0015] Specifically, Step 2 is as follows:

[0016] The "road surface completion" image processing technology is written in Python with the cv2 module imported. It can automatically and batch-replace the content in the bounding boxes of manually annotated road surface defect datasets with normal road surface content. Specifically:

[0017] Step 2.1: Use the unprocessed, manually labeled images of road surface defects as the images to be processed;

[0018] Step 2.2: Based on the file name of the image to be processed, find the corresponding YOLO format label file, obtain the label data rows corresponding to the bounding boxes in order from top to bottom, and calculate the center point position, length and width pixel count, and area occupied by all bounding boxes of the bounding boxes;

[0019] Step 2.3: Based on the area occupied by all the bounding boxes, rank them and find the bounding box with the largest area, and record the row number of the corresponding label data;

[0020] Step 2.4: Select a normal road surface image with the same dimensions as the image to be processed, use it as the background image, and adjust its grayscale value so that the grayscale value of the normal road surface image reaches the average grayscale value of the image to be processed.

[0021] Step 2.5: Based on the calculated center point position, length and width pixel count, and area occupied by all bounding boxes, replace the pixels in the bounding boxes point-to-point with the corresponding pixel values ​​in the background image to obtain a new background image;

[0022] Step 2.6: Set the length of the pixels that need to be Gaussian blurred in two opposite directions perpendicular to the edge of the new background image region to X. Apply Gaussian blur to the edge of the new background image region X pixels inward and outward respectively. The distance between the inward blur boundary and the outward blur boundary is 2X pixels.

[0023] Step 2.7: Define the images processed by Step 2.1-Step 2.6 as road surface completion images, and use them as positive samples in the missed detection model dataset. Insert the two characters "P_" at the beginning of the file name, where "P" indicates a positive sample and "_" is used to separate the subsequent file names. Finally, store the positive samples in the missed detection model positive sample subfolder under the corresponding disease folder.

[0024] Step 2.8: Extract the content of the bounding box with the largest area from the image to be processed, and restore it point-to-point to the road surface completion image to obtain a negative sample of the missed detection model dataset that retains the largest defect and removes all other defects. Insert the two characters "N_" at the beginning of the file name, where "N" represents a negative sample and "_" is used to separate the subsequent file names. Finally, store the negative sample in the missed detection model negative sample subfolder under the corresponding defect folder.

[0025] Step 2.9: Return to Step 2.2 and obtain the label data row corresponding to the new boundingbox; if all label data rows contained in the image to be processed have been processed, return to Step 2.1; if the manually labeled data dataset of road surface defects has been processed, the "road surface completion" image processing technology will end, and the positive and negative samples in the missed detection model dataset will be completed.

[0026] Step 2.10: Divide the positive and negative samples under the six disease folders into a 5:1 ratio to obtain the training set and validation set of the missed detection model dataset.

[0027] Specifically, Step 3 is as follows:

[0028] Step 3.1: Train a missed detection model based on the YOLOv5 classification model using the training set;

[0029] Step 3.2: Use the missed detection model to perform binary classification missed detection on the test set to obtain the missed detection results;

[0030] Step 3.3: Compare the category of the missed detection result with the first character of the road surface completion image file name. If they match, the missed detection model is judged to be correctly classified; otherwise, the missed detection model is judged to be incorrectly classified, and the incorrectly classified image is judged a second time, including manual judgment and reprocessing using the "road surface completion" image processing technology, and then judging again using the missed detection model.

[0031] Specifically, Step 4 is as follows:

[0032] Step 4.1: When a road surface completion image is classified as a positive sample, there are no cases of missed defects in the corresponding manually labeled road surface defect dataset images.

[0033] Step 4.2: When a road surface completion image is classified as a negative sample, the corresponding manually labeled road surface defect dataset image may have missed defects, and the next step of judgment is carried out.

[0034] Specifically, the next step of the determination is as follows:

[0035] The system manually reconfirms whether any defects were missed. If none were missed, the manually annotated pavement defect dataset images are reclassified as defect-free. If defects were missed, the system continues to determine whether bounding boxes need to be drawn and labels added for the missed defects. If so, object detection and annotation tools, such as labelimg, are used for supplementary labeling. If not, no action is taken on the manually annotated pavement defect dataset images.

[0036] The beneficial effects of this invention are:

[0037] This invention uses a missed detection model (binary classification) based on the YOLOv5 classification model to detect whether there are large and obvious undetected defects in the hand-marked pavement defect dataset. It can automatically and in batches determine whether there are missed detection issues in the hand-marked dataset after "pavement completion" image processing technology with high accuracy. This invention replaces manual judgment with a model for missed detection; subsequent checks only require reviewing all result images classified as negative samples, saving labor and time costs. It also improves the quality of the pavement defect hand-marked dataset and enhances the quality of the pavement defect target detection result images. Attached Figure Description

[0038] Figure 1 This is a flowchart of the steps of the present invention;

[0039] Figure 2 This is a simplified flowchart of the road surface repair method of the present invention;

[0040] Figure 3 This is a comparison diagram showing the positive and negative sample processing results of the road surface completion method of the present invention;

[0041] Figure 4 This diagram illustrates the method by which the present invention determines whether the classification results of the missed detection model are correct.

[0042] Figure 5 The above is a test case diagram of the positive and negative sample model classification of this invention;

[0043] Figure 6 The following is a case study of the positive and negative sample model classification of this invention (below);

[0044] Figure 7 This is a diagram showing the detection results of the missed detection model of this invention on two test sets;

[0045] Figure 8 This is a graph showing the evaluation index of the missed detection model of this invention;

[0046] Figure 9 This is a processing and result display diagram of missed images of six different diseases according to the present invention. Detailed Implementation

[0047] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0048] Example 1: As Figure 1 As shown, a method for detecting missed defects in pavement damage targets includes the following steps:

[0049] Step 1: Collect manually labeled dataset images of six different types of road surface defects with the same quantity;

[0050] Step 2: Use the "road surface completion" image processing technology to convert the manually labeled road surface defect dataset into a binary classification dataset;

[0051] Step 3: Train the missed detection model using a binary classification dataset, and perform missed detection to obtain the missed detection results;

[0052] Step 4: Analyze the missed detection results. If there are any missed detections, use the target detection and annotation tool to supplement and mark the missed road surface defects.

[0053] The specific steps of Step 1 are as follows:

[0054] Step 1.1: Obtain manually labeled dataset images (resolution 2048*2048) of 40 images each of six different types of pavement defects, and store them in the image subfolders of six different folders. The file names of the six different defects are LIE_FENG, JUN_LIE, FAN_YOU, KENG_CAO, SONG_SAN, and XIU_BU, which are the uppercase pinyin letters of the six defects, and the same applies below;

[0055] Step 1.2: Based on the file names of the dataset images collected in Step 1.1, find the corresponding label files and store them uniformly in the label subfolders under the six folders;

[0056] Step 1.3: Use an object detection and annotation tool (such as labelimg) to check the annotation status of the collected road surface defect manual annotation dataset to ensure that there are no missing annotations.

[0057] The specific steps of Step 2 are as follows:

[0058] The "road surface completion" image processing technology is written in Python with the cv2 module imported. It can automatically and batch-replace the content in the bounding boxes of manually annotated road surface defect datasets with normal road surface. A simplified flowchart of the road surface completion method is shown below. Figure 2 As shown, specifically:

[0059] Step 2.1: Use the unprocessed manually labeled images of the road surface defects dataset as the images to be processed. The resolution of the images to be processed is 1500*1500.

[0060] Step 2.2: Based on the file name of the image to be processed, find the corresponding YOLO format label file, obtain the label data rows corresponding to the bounding boxes in order from top to bottom, and calculate the center point position, length and width pixel count, and area occupied by all bounding boxes of the bounding boxes;

[0061] Step 2.3: Based on the area occupied by all the bounding boxes, rank them and find the bounding box with the largest area, and record the row number of the corresponding label data;

[0062] Step 2.4: Select a normal road surface image with the same dimensions as the image to be processed, use it as the background image, with a resolution of 1500*1500, and adjust the grayscale value so that the grayscale value of the normal road surface image reaches the average grayscale value of the image to be processed.

[0063] Step 2.5: Based on the calculated center point position, length and width pixel count, and area occupied by all bounding boxes, replace the pixels in the bounding boxes point-to-point with the corresponding pixel values ​​in the background image to obtain a new background image;

[0064] Step 2.6: Apply Gaussian blur to the edges of all image regions replaced in Step 2.5. Set the pixel length to be blurred in two opposite directions perpendicular to the edge of the replaced image region to 50. The edges of the replaced image regions will be Gaussian blurred 50 pixels inward and outward respectively, with the distance between the inward and outward blur boundaries being 100 pixels each. Apply Gaussian blur to all image regions replaced in Step 2.5 and the fixed 50-pixel region outside the edges to obtain a region-blurred image. The region-blurred image is based on the original bounding box area size, with 50 pixels calculated outward from each of the four edges (top, bottom, left, and right), and then Gaussian blurred on the resulting region (if 50 pixels outward from the edge exceeds the image size, the excess part will be discarded, and only the pixels up to the image edge will be retained). Then, replace the sub-regions 100 pixels inward from the edge of each blurred region point-to-point with the pixel values ​​of the background image;

[0065] Step 2.7: Define the images processed by Step 2.1-Step 2.6 as road surface completion images, and use them as positive samples in the missed detection model dataset. Insert the two characters "P_" at the beginning of the file name, where "P" represents a positive sample and "_" is used to separate the subsequent file names. Finally, store the positive samples in the positive_samples subfolder of the missed detection model under the corresponding disease folder.

[0066] Step 2.8: Extract the content of the largest bounding box from the image to be processed and restore it point-to-point into the road surface completion image. This yields a negative sample dataset of the missed detection model, which retains the largest defect and removes all other defects. Insert "N_" at the beginning of the filename, where "N" represents a negative sample and "_" separates subsequent filenames. Finally, store the negative samples in the "negative_samples" subfolder under the corresponding defect folder. A comparison of some positive and negative sample processing results of the road surface completion method is shown below. Figure 3 As shown.

[0067] Step 2.9: Return to Step 2.2 and obtain the label data row corresponding to the new boundingbox; if all label data rows contained in the image to be processed have been processed, return to Step 2.1; if the manually labeled data dataset of road surface defects has been processed, the "road surface completion" image processing technology will end, and the positive and negative samples in the missed detection model dataset will be completed.

[0068] Step 2.10: Divide the positive and negative samples under the six disease folders into a 5:1 ratio to obtain the training set and validation set of the missed detection model dataset, and store them in the subfolders positive and negative under the training and validation folders of the missed detection model dataset, respectively.

[0069] The specific steps of Step 3 are as follows:

[0070] Step 3.1: Train the missed detection model using the training set. The main hyperparameter settings are shown in Table 1.

[0071] Table 1 Hyperparameter Settings

[0072] Number of training epochs 100 Batch size 20 Image size (imgsz) 1024*1024 Number of threads (workers) 4 Learning rate 0.001 Optimizer Adam Weight decay 5.0e-05

[0073] Step 3.2: This invention designs two test sets. The first test set is obtained by uniformly rotating the validation set 90° clockwise, with 40 positive and 40 negative samples each. The second test set is created from a second collection of hand-labeled datasets of six diseases, with 120 positive and 120 negative samples each. A missed detection model is used to perform binary classification detection on both test sets.

[0074] Step 3.3: Compare the category of the missed detection result with the first character of the road surface completion image file name. If they match, the missed detection model is considered correctly classified; otherwise, the missed detection model is considered incorrectly classified, and the incorrectly classified image undergoes a second review, including manual review and reprocessing using the "road surface completion" image processing technology before re-applying the missed detection model. The method for determining whether the classification result of the missed detection model is correct is as follows: Figure 4 As shown; a case study of model classification for positive and negative samples. Figure 5 and Figure 6 As shown.

[0075] Experimental results show that the missed detection model achieved classification accuracy of 88% and 90% on the first and second test sets, respectively, with precision of 0.86 and 0.859, recall of 0.925 and 0.966, and F1 scores of 0.89 and 0.9, demonstrating its ability to automatically classify missed detections. The detection results (confusion matrix) of the missed detection model on the two test sets are shown below. Figure 7 As shown, the model evaluation metrics are as follows: Figure 8 As shown.

[0076] The specific steps of Step 4 are as follows:

[0077] Step 4.1: When a road surface completion image is classified as a positive sample (the category with the highest probability is P), then there are no cases of missed defects in the corresponding manually labeled road surface defect dataset images.

[0078] Step 4.2: When a road surface completion image is classified as a negative sample (the category with the highest probability is N), the corresponding road surface defect manually labeled dataset image may have missed defects, and the next step of judgment is carried out.

[0079] The next step is as follows: Manually reconfirm whether any defects were missed. If not, re-determine that the manually labeled pavement defect dataset images are free of defects. If defects were missed, determine whether bounding boxes need to be drawn and labels added for the missed defects. If so, use the object detection and labeling tool `labelimg` for supplementary labeling; otherwise, do not perform any operations on the manually labeled pavement defect dataset images. The partial processing results for six different defect types are shown below. Figure 9 As shown.

[0080] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A method for detecting missed detections of pavement defects, characterized in that, The specific steps of the method are as follows: Step 1: Collect manually labeled data sets of road surface defects; Step 2: Use "road surface completion" image processing technology to convert the manually labeled road surface defect dataset into a binary classification dataset; Step 3: Train the missed detection model using a binary classification dataset, and perform missed detection to obtain the missed detection results; Step 4: Analyze the results of missed detections. If there are any missed detections, use the target detection and annotation tool to mark the missed road surface defects. Step 2 specifically refers to: The image processing technology for "road surface completion" automatically and in batches replaces the contents of the bounding boxes in the manually annotated road surface defect dataset with normal road surface data. Specifically: Step 2.1: Use the unprocessed, manually labeled images of road surface defects as the images to be processed; Step 2.2: Based on the file name of the image to be processed, find the corresponding YOLO format label file, obtain the label data rows corresponding to the bounding box in order from top to bottom, and calculate the center point position, length and width pixel count, and area occupied by all bounding boxes of the bounding box; Step 2.3: Based on the area occupied by all the bounding boxes, rank them and find the bounding box with the largest area, and record the row number of the corresponding label data; Step 2.4: Select a normal road surface image with the same dimensions as the image to be processed, use it as the background image, and adjust its grayscale value so that the grayscale value of the normal road surface image reaches the average grayscale value of the image to be processed. Step 2.5: Based on the calculated center point position, length and width pixel count, and area occupied by all bounding boxes, replace the pixels in the bounding boxes point-to-point with the corresponding pixel values ​​in the background image to obtain a new background image; Step 2.6: Set the length of the pixels that need to be Gaussian blurred in two opposite directions perpendicular to the edge of the new background image region to X. Apply Gaussian blur to the edge of the new background image region X pixels inward and outward respectively. The distance between the inward blur boundary and the outward blur boundary is 2X pixels. Step 2.7: Define the images processed by Step 2.1-Step 2.6 as road surface completion images, and use them as positive samples in the missed detection model dataset. Insert the two characters "P_" at the beginning of the file name, where "P" indicates a positive sample and "_" is used to separate the subsequent file names. Finally, store the positive samples in the missed detection model positive sample subfolder under the corresponding disease folder. Step 2.8: Extract the content of the bounding box with the largest area from the image to be processed, and restore it point-to-point to the road surface completion image to obtain a negative sample of the missed detection model dataset that retains the largest defect and removes all other defects. Insert the two characters "N_" at the beginning of the file name, where "N" represents a negative sample and "_" is used to separate the subsequent file names. Finally, store the negative sample in the missed detection model negative sample subfolder under the corresponding defect folder. Step 2.9: Return to Step 2.2 and obtain the label data row corresponding to the new bounding box; if all label data rows in the image to be processed have been processed, return to Step 2.1; if the manually labeled data dataset of road surface defects has been processed, the "road surface completion" image processing technology will end, and the positive and negative samples in the missed detection model dataset will be completed. Step 2.10: Divide the positive and negative samples under the six disease folders into a uniform proportion to obtain the training set and validation set of the missed detection model dataset.

2. The method for detecting missed detections of pavement defects according to claim 1, characterized in that, Step 1 specifically refers to: Step 1.1: Obtain the dataset of manually labeled road surface defects. The images are all grayscale images collected by the inspection vehicle and stored in the image subfolders of the six folders respectively. Step 1.2: Based on the file names of the manually labeled pavement defects dataset, find the corresponding label files and store them uniformly in the label subfolders under the six folders; Step 1.3: Use the object detection and annotation tool to check the annotation status of the manually annotated dataset of road surface defects to ensure that there are no missing annotations.

3. The method for detecting missed detections of pavement defects according to claim 1, characterized in that, Step 3 specifically refers to: Step 3.1: Train the missed detection model using the training set; Step 3.2: Use the missed detection model to perform binary classification missed detection on the test set to obtain the missed detection results; Step 3.3: Compare the category of the missed detection result with the first character of the road surface completion image file name. If they match, the missed detection model is determined to be correctly classified. Otherwise, the missed detection model is judged to be misclassified, and the misclassified images are subject to secondary judgment, including manual judgment and reprocessing using the "road surface completion" image processing technology, and then the missed detection model is used again for judgment.

4. The method for detecting missed detections of pavement defects according to claim 1, characterized in that, Step 4 specifically refers to: Step 4.1: When a road surface completion image is classified as a positive sample, there are no cases of missed defects in the corresponding manually labeled road surface defect dataset images. Step 4.2: When a road surface completion image is classified as a negative sample, the corresponding manually labeled road surface defect dataset image may have missed defects, and the next step of judgment is carried out.

5. A method for detecting missed detections of pavement defects according to claim 4, characterized in that, The next step in the determination is as follows: The system manually reconfirms whether any defects have been missed. If none are found, the manually labeled pavement defect dataset images are reclassified as defect-free. If defects are found, the system continues to determine whether bounding boxes need to be drawn and labels added for the missed defects. If so, the objects are labeled using an object detection tool. If not, no action is taken on the manually labeled pavement defect dataset images.

Images