Leakage detection method for pavement disease target detection

Abstract

The invention relates to a leak detection method for pavement disease target detection, and belongs to the field of highway pavement disease detection and quality control. The invention provides a'pavement completion 'image processing technology, any bounding box can be uniformly replaced with a normal pavement, meanwhile, a data set of a leak detection model is made, and the leak detection model is trained. The leak detection model can automatically detect whether a large and obvious disease leak detection problem exists in an area except for the bounding box in the picture, and prerequisite preparation is made for subsequent disease supplementary marking work. The method provided by the invention can automatically improve the disease missing detection condition in the pavement disease target detection result picture.

Description

Technical Field

[0001] The invention relates to a missed detection method for road surface disease target detection, and belongs to the field of highway road surface disease detection and quality control. Background Art

[0002] Asphalt pavement is the most widely used advanced pavement in road construction. With the development of China's self-produced road asphalt material industry, asphalt pavement has been widely used in urban roads and highway trunk lines, becoming the most paved advanced pavement in China. However, with the increase in traffic volume, frequent weather changes and other factors, various pavement diseases continue to appear, and the types of pavement diseases include but are not limited to: cracks, repairs and potholes. The work of the present invention involves six different pavement diseases, namely cracks, cracks, looseness, potholes, oil spills and repairs. At present, the mainstream method used in the computer field to detect pavement diseases is target detection in deep learning and other methods, and the pavement disease target detection result image obtained by this method will inevitably have a small number of disease missed detection problems. At present, the solutions to the missed detection problem mainly include manual review and model-based review: manual review may consume a lot of time and human resources when the number of target detection result images is large; although the model-based review theoretically consumes less resources than the manual method, there is no mature solution example at present, so the effect is unknown.

[0003] The only feasible way to check for missed detection is to manually review the target detection result images one by one, that is, to check whether there are any misdetected disease types within the bounding box, and whether there are any diseases that are missed outside the bounding box. After multiple reviews and corrections, a more accurate target detection result image can be obtained. However, the current method has many shortcomings, and the problems are as follows: First, the human and material resources consumed generally increase with the increase in the number of target detection result images; second, the review will take a lot of extra time, and this time is difficult to reduce; third, long-term boring and repetitive work will make the reviewers lack energy and attention, which may lead to low review efficiency and substandard quality; fourth, the annotation level of different reviewers is inconsistent and subjective, which may lead to inconsistent results after the review. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a missed detection method for pavement disease target detection, aiming to solve the problem that there are a small number of missed detections in the pavement disease target detection result map, and currently only manual review can be used for detection, which has the technical problems of large resource consumption, high time cost, low efficiency caused by subjective labeling, and inconsistent results.

[0005] The technical solution of the present invention is: the present invention designs a data set preprocessing method called "road completion" image processing technology, which can uniformly process the hand-marked data set of road surface diseases into the data set of missed detection model. The missed detection model (two-classification) can be trained using this data set. The model can detect whether there are still large and obvious diseases in any road surface completion picture. If it is detected as a positive sample, there is no missed detection problem; if it is detected as a negative sample, there is a missed detection problem, and it is necessary to use the target detection and annotation tool to supplement the disease annotation. The present invention attempts to fill the gap in the field of using models to conduct missed detection review work on target detection result images, and has the potential to automatically and batch detect missed detection problems.

[0006] A missed detection method for road surface disease target detection, the specific steps are:

[0007] Step 1: Collect manually annotated data sets of pavement defects;

[0008] Step 2: Use the “pavement completion” image processing technology to convert the manually labeled pavement disease dataset into a binary classification dataset;

[0009] Step 3: Use the binary classification data set to train the missed detection model and perform missed detection to obtain the missed detection results;

[0010] Step 4: Analyze the missed detection results. If there are any missed detections, use the target detection and annotation tools to add additional marks to the missed road defects.

[0011] Specifically, the Step 1 is as follows:

[0012] Step 1.1: Obtain six different types of manually annotated data sets of pavement defects with the same number. All images are grayscale images collected by the inspection vehicle. The image size is 2048*2048 and they are stored in the image subfolders of six folders respectively.

[0013] Step 1.2: According to the file name of the dataset of manually annotated pavement defects, find the corresponding label file and store it in the label subfolder under the six folders;

[0014] Step 1.3: Use the target detection and annotation tool to check the annotation status of the collected manually annotated data set of pavement defects to ensure that there are no missed labels.

[0015] Specifically, the Step 2 is as follows:

[0016] The “road completion” image processing technology is written in Python with the cv2 module imported. It can automatically and batch-replace the content in the bounding box of the pavement disease manual annotation dataset with normal pavement. Specifically:

[0017] Step 2.1: Take the unprocessed pavement disease manually annotated dataset images as the images to be processed;

[0018] Step 2.2: According to the file name of the image to be processed, find the corresponding label file in YOLO format, obtain the label data row corresponding to the bounding box in order from top to bottom in units of rows, and calculate the center point position, length and width pixel number and area occupied by all bounding boxes of the bounding box;

[0019] Step 2.3: According to the area occupied by all the bounding boxes, sort out the bounding boxes with the largest area and record the row number of the corresponding label data;

[0020] Step 2.4: Select a normal road surface image with the same length and width as the image to be processed as the background image, and adjust the grayscale value so that the grayscale value of the normal road surface image reaches the average grayscale value of the image to be processed;

[0021] Step 2.5: According to the calculated center point position, length and width pixel number and area occupied by all bounding boxes of the bounding box, replace the pixels in the bounding box with the corresponding pixel values ​​in the background image point by point to obtain a new background image;

[0022] Step 2.6: Set the pixel length of the new background image area in two opposite directions perpendicular to the edge and requiring Gaussian blur to be X, and Gaussian blur the edge of the new background image area X pixels inward and outward respectively, and the distance between the inward blur boundary and the outward blur boundary is 2X pixels;

[0023] Step 2.7: Define the image processed by Step 2.1-Step 2.6 as a pavement completion image as a positive sample of the missed detection model data set, and insert two characters "P_" at the beginning of the file name, where "P" represents a positive sample and "_" is used to separate subsequent file names. Finally, the positive sample is stored in the missed detection model positive sample subfolder under the corresponding disease folder;

[0024] Step 2.8: Cut out the bounding box with the largest area from the image to be processed, and restore it point by point to the road completion image to obtain a negative sample of the missed detection model data set that retains the largest defect and removes all other defects. Insert two characters "N_" at the beginning of the file name, where "N" represents the negative sample and "_" is used to separate subsequent file names. Finally, store the negative sample in the missed detection model negative sample subfolder under the corresponding defect folder.

[0025] Step 2.9: Return to Step 2.2 and obtain the label data row corresponding to the new bounding box; if all the label data rows contained in the image to be processed have been processed, return to Step 2.1; if all the pavement disease manual annotation datasets have been processed, the "pavement completion" image processing technology is executed, and the positive and negative samples in the missed detection model dataset are produced;

[0026] Step 2.10: Divide the positive and negative samples in the six disease folders into a uniform ratio of 5:1 to obtain the training set and validation set of the missed detection model data set.

[0027] Specifically, the Step 3 is as follows:

[0028] Step 3.1: Use the training set to train the missed detection model based on the yolov5 classification model;

[0029] Step 3.2: Use the missed detection model to perform binary missed detection on the test set to obtain the missed detection results;

[0030] Step 3.3: Compare the category of the missed detection result and the first character of the road surface completion image file name. If the two are consistent, the missed detection model classification is determined to be correct; otherwise, the missed detection model classification is determined to be wrong, and the wrongly classified image is subjected to secondary judgment, including manual judgment and re-processing using the "road surface completion" image processing technology, and then judgment is made again using the missed detection model.

[0031] Specifically, the Step 4 is as follows:

[0032] Step 4.1: When a pavement completion image is classified as a positive sample, the corresponding pavement disease manual annotation dataset image has no disease missed detection;

[0033] Step 4.2: When a road surface completion image is classified as a negative sample, the corresponding road surface disease manual annotation dataset image may have missed diseases, and the next step of judgment is carried out.

[0034] Specifically, the next step is to determine:

[0035] Manually confirm again whether there are any missed detections. If not, re-judge the pavement disease manual annotation dataset image as a case of no missed detection of diseases. If so, continue to determine whether it is necessary to draw bounding boxes and supplement labels for the missed diseases. If necessary, use target detection and annotation tools such as labelimg to perform supplementary marking. If not, do not perform any operation on the pavement disease manual annotation dataset image.

[0036] The beneficial effects of the present invention are:

[0037] The present invention uses a missed detection model (two-classification) based on the yolov5 classification model to detect whether there are large and obvious diseases missed in the pavement disease hand-marked data set, and can automatically and batch-wise determine whether there are missed detection problems in the hand-marked data set processed by the "pavement completion" image processing technology with a high accuracy rate. The present invention uses a model to replace manual missed detection judgment work, and only needs to review all result images classified as negative samples later, which can save labor and time costs, and at the same time can improve the quality of the pavement disease hand-marked data set and the quality of the pavement disease target detection result images. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a flow chart of the steps of the present invention;

[0039] Figure 2 is a simplified flow chart of the road surface completion method of the present invention;

[0040] Figure 3 This is a comparative display diagram of some positive and negative sample processing results of the road surface completion method of the present invention;

[0041] Figure 4 It is a diagram of the method of judging whether the classification result of the missed detection model of the present invention is correct;

[0042] Figure 5 : is a diagram of a measured case of model classification of positive and negative samples of the present invention (above);

[0043] Figure 6 This is a diagram of a measured case of model classification of positive and negative samples of the present invention (below);

[0044] Figure 7 It is a display diagram of the detection results of the missed detection model of the present invention on two test sets;

[0045] Figure 8 is an evaluation index diagram of the missed detection model of the present invention;

[0046] Fig. 9 It is a diagram showing the processing and result display of missed detection images of six different diseases by the present invention. DETAILED DESCRIPTION

[0047] The present invention will be further described below in conjunction with the accompanying drawings and specific implementation methods.

[0048] Example 1: Figure 1 As shown in FIG. 1 , a missed detection method for road surface disease target detection is shown in FIG. 1 , and the specific steps are as follows:

[0049] Step 1: Collect six different types of pavement disease manually annotated dataset images with the same number;

[0050] Step 2: Use the “pavement completion” image processing technology to convert the manually labeled pavement disease dataset into a binary classification dataset;

[0051] Step 3: Use the binary classification data set to train the missed detection model and perform missed detection to obtain the missed detection results;

[0052] Step 4: Analyze the missed detection results. If there are any missed detections, use the target detection and annotation tools to add additional marks to the missed road defects.

[0053] The specific steps of Step 1 are:

[0054] Step 1.1: Obtain six different types of 40 manually annotated pavement disease dataset images (resolution 2048*2048), and store them in the image subfolder of six folders. The file names of the six different diseases are LIE_FENG, JUN_LIE, FAN_YOU, KENG_CAO, SONG_SAN, XIU_BU, which are the capital pinyin letters of the six diseases, the same below;

[0055] Step 1.2: According to the file names of the dataset images collected in Step 1.1, find the corresponding label files and store them in the label subfolders under the six folders;

[0056] Step 1.3: Use target detection and annotation tools (such as labelimg) to check the annotation status of the collected pavement disease hand-annotated dataset to ensure that there are no missing labels.

[0057] The specific steps of Step 2 are:

[0058] The image processing technology of "road completion" is written in Python language with cv2 module imported. It can automatically and batch replace the content in the bounding box of the manually annotated road disease dataset with normal road surface. The brief flow chart of the road completion method is as follows: Figure 2 As shown, specifically:

[0059] Step 2.1: Use the unprocessed pavement disease manually annotated dataset images as the images to be processed, with a resolution of 1500*1500;

[0060] Step 2.2: According to the file name of the image to be processed, find the corresponding label file in YOLO format, obtain the label data row corresponding to the bounding box in order from top to bottom in units of rows, and calculate the center point position, length and width pixel number and area occupied by all bounding boxes of the bounding box;

[0061] Step 2.3: According to the area occupied by all the bounding boxes, sort out the bounding boxes with the largest area and record the row number of the corresponding label data;

[0062] Step 2.4: Select a normal road surface picture with the same length and width as the picture to be processed as the background picture, with a resolution of 1500*1500, and adjust the grayscale value so that the grayscale value of the normal road surface picture reaches the average grayscale value of the picture to be processed;

[0063] Step 2.5: According to the calculated center point position, length and width pixel number and area occupied by all bounding boxes of the bounding box, replace the pixels in the bounding box with the corresponding pixel values ​​in the background image point by point to obtain a new background image;

[0064] Step2.6: Perform Gaussian blur operation on the edges of all image areas replaced in Step2.5, set the pixel length to be blurred in two opposite directions perpendicular to the edge of the replaced image area to 50, and Gaussian blur the edge of the replaced image area 50 pixels inward and outward respectively, and the distance of the blurred boundary inward and outward is 100 pixels. Perform Gaussian blur operation on all image areas replaced in Step2.5 and the fixed 50 pixel areas outside their edges to obtain regional blurred images. The regional blurred image is based on the size of the original boundingbox area, and the four edges of the top, bottom, left and right are calculated 50 pixels outward respectively, and then the obtained area is Gaussian blurred (if 50 pixels from the edge to the outside exceeds the size of the image, the excess part will be discarded and only the pixels to the edge of the image will be retained). Then, the sub-areas of 100 pixels inward from the edge of each blurred area are replaced point by point with the pixel values ​​of the background image;

[0065] Step 2.7: Define the image processed by Step 2.1-Step 2.6 as a pavement completion image as a positive sample of the missed detection model data set, and insert two characters "P_" at the beginning of the file name, where "P" represents a positive sample and "_" is used to separate subsequent file names. Finally, store the positive sample in the missed detection model positive sample subfolder positive_samples under the corresponding disease folder;

[0066] Step 2.8: The bounding box with the largest area is cropped from the image to be processed, and restored point by point to the road completion image to obtain a negative sample of the missed detection model data set that retains the largest defect and removes all other defects. The two characters "N_" are inserted at the beginning of the file name, where "N" represents the negative sample and "_" is used to separate subsequent file names. Finally, the negative sample is stored in the missed detection model negative sample subfolder negative_samples under the corresponding defect folder; the comparison of some positive and negative sample processing results of the road completion method is shown as follows Figure 3 shown.

[0067] Step 2.9: Return to Step 2.2 and obtain the label data row corresponding to the new bounding box; if all the label data rows contained in the image to be processed have been processed, return to Step 2.1; if all the pavement disease manual annotation datasets have been processed, the "pavement completion" image processing technology is executed, and the positive and negative samples in the missed detection model dataset are produced;

[0068] Step 2.10: Divide the positive and negative samples in the six disease folders in a uniform ratio of 5:1 to obtain the training set and verification set of the missed detection model data set, and store them in the subfolders positive and negative under the training and verification folders of the missed detection model data set, respectively.

[0069] The specific steps of Step 3 are:

[0070] Step 3.1: Use the training set to train the missed detection model. The main hyperparameter settings are shown in Table 1:

[0071] Table 1 Hyperparameter settings

[0072] Hyperparameters parameter Number of training rounds (epochs) 100 Batch size (batch_size) 20 Image size (imgsz) 1024*1024 Number of multithreaded workers 4 Learning rate 0.001 Optimizer Adam Weight decay (Decay) 5.0e-05

[0073] Step 3.2: The present invention designs two test sets. The first test set is obtained by rotating the validation set clockwise by 90°, with 40 positive and negative samples each; the second test set is made from the six disease hand-labeled data sets collected twice, with 120 positive and negative samples each. The missed detection model is used to perform binary classification detection on the two test sets.

[0074] Step 3.3: Compare the category of the missed detection result and the first character of the road completion image file name. If the two are consistent, the missed detection model classification is determined to be correct; otherwise, the missed detection model classification is determined to be wrong, and the wrongly classified image is subjected to secondary judgment, including manual judgment and re-processing using the "road completion" image processing technology, and then re-judged using the missed detection model. The method for judging whether the classification result of the missed detection model is correct is as follows: Figure 4 As shown; the actual test cases of model classification of positive and negative samples are as follows Figure 5 and Figure 6 shown.

[0075] The experimental results show that the missed detection model achieved 88% and 90% classification accuracy in the first and second test sets, respectively, with precision of 0.86 and 0.859, recall of 0.925 and 0.966, and F1 of 0.89 and 0.9, which can realize automatic classification of missed detection. The detection results (confusion matrix) of the missed detection model on the two test sets are shown in the figure below. Figure 7 As shown in Figure 8 shown.

[0076] The specific steps of Step 4 are:

[0077] Step 4.1: When a pavement completion image is classified as a positive sample (the category corresponding to the maximum probability is P), the corresponding pavement disease manual annotation dataset image has no disease missed detection;

[0078] Step 4.2: When a road surface completion image is classified as a negative sample (the category corresponding to the maximum probability is N), the corresponding road surface disease manual annotation dataset image may have missed diseases, and the next step of judgment is carried out.

[0079] The next step is to manually confirm whether there is any missed detection. If not, re-determine whether the pavement disease manual annotation dataset image has no missed detection; if so, continue to determine whether it is necessary to draw bounding boxes and add labels for the missed diseases. If necessary, use the target detection and annotation tool labelimg to add additional labels. If not, do not do anything with the pavement disease manual annotation dataset image. The partial processing results of the present invention for six different types of diseases are shown as follows: Fig. 9 shown.

[0080] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A missed detection method for road surface disease target detection, characterized in that: The specific steps of the method are as follows: Step 1: Collect manually annotated data sets of pavement defects; Step 2: Use the "pavement completion" image processing technology to convert the pavement disease manual annotation dataset into a binary classification dataset; Step 3: Use the binary classification data set to train the missed detection model and perform missed detection to obtain the missed detection results; Step 4: Analyze the missed detection results. If there are any missed detections, use the target detection and annotation tools to add additional marks to the missed road defects.

2. The missed detection method for road surface disease target detection according to claim 1, characterized in that: The Step 1 is specifically as follows: Step 1.1: Obtain a dataset of manually annotated road surface defects. All images are grayscale images collected by the inspection vehicle and are stored in the image subfolders of six folders. Step 1.2: According to the file name of the dataset of manually annotated pavement defects, find the corresponding label file and store it in the label subfolder under the six folders; Step 1.3: Use the target detection and annotation tool to check the annotation status of the collected manually annotated data set of pavement defects to ensure that there are no missed labels.

3. The missed detection method for road surface disease target detection according to claim 1, characterized in that: The Step 2 is specifically as follows: The image processing technology of "road completion" can automatically and batch replace the content in the bounding box of the road disease manual annotation dataset with normal road surface, specifically: Step 2.1: Take the unprocessed pavement disease manually annotated dataset images as the images to be processed; Step 2.2: According to the file name of the image to be processed, find the corresponding label file in YOLO format, obtain the label data row corresponding to the bounding box in order from top to bottom in units of rows, and calculate the center point position, length and width pixel number and area occupied by all bounding boxes of the bounding box; Step 2.3: According to the area occupied by all the bounding boxes, sort out the bounding boxes with the largest area and record the row number of the corresponding label data; Step 2.4: Select a normal road surface image with the same length and width as the image to be processed as the background image, and adjust the grayscale value so that the grayscale value of the normal road surface image reaches the average grayscale value of the image to be processed; Step 2.5: According to the calculated center point position, length and width pixel number and area occupied by all bounding boxes of the bounding box, replace the pixels in the bounding box with the corresponding pixel values ​​in the background image point by point to obtain a new background image; Step 2.6: Set the pixel length of the new background image area in two opposite directions perpendicular to the edge and requiring Gaussian blur to be X, and Gaussian blur the edge of the new background image area X pixels inward and outward respectively, and the distance between the inward blur boundary and the outward blur boundary is 2X pixels; Step 2.7: Define the image processed by Step 2.1-Step 2.6 as a pavement completion image as a positive sample of the missed detection model data set, and insert two characters "P_" at the beginning of the file name, where "P" represents a positive sample and "_" is used to separate subsequent file names. Finally, the positive sample is stored in the missed detection model positive sample subfolder under the corresponding disease folder; Step 2.8: Cut out the bounding box with the largest area from the image to be processed, and restore it point by point to the road completion image to obtain a negative sample of the missed detection model data set that retains the largest defect and removes all other defects. Insert two characters "N_" at the beginning of the file name, where "N" represents the negative sample and "_" is used to separate subsequent file names. Finally, store the negative sample in the missed detection model negative sample subfolder under the corresponding defect folder. Step 2.9: Return to Step 2.2 and obtain the label data row corresponding to the new bounding box; if all the label data rows contained in the image to be processed have been processed, return to Step 2.1; if all the pavement disease manual annotation datasets have been processed, the "pavement completion" image processing technology is executed, and the positive and negative samples in the missed detection model dataset are produced; Step 2.10: Divide the positive and negative samples in the six disease folders in a uniform proportion to obtain the training set and validation set of the missed detection model data set.

4. The missed detection method for road surface disease target detection according to claim 1, characterized in that: The Step 3 is specifically as follows: Step 3.1: Use the training set to train the missed detection model; Step 3.2: Use the missed detection model to perform binary missed detection on the test set to obtain the missed detection results; Step 3.3: Compare the category of the missed detection result and the first character of the road surface completion image file name. If the two are consistent, it is determined that the missed detection model classification is correct; Otherwise, the missed detection model is judged to have made a classification error, and the incorrectly classified images are subjected to a second judgment, including manual judgment and reprocessing using the "road completion" image processing technology, and then the missed detection model is used for judgment again.

5. The missed detection method for road surface disease target detection according to claim 1, characterized in that: The Step 4 is specifically as follows: Step 4.1: When a pavement completion image is classified as a positive sample, the corresponding pavement disease manual annotation dataset image has no disease missed detection; Step 4.2: When a road surface completion image is classified as a negative sample, the corresponding road surface disease manual annotation dataset image may have missed diseases, and the next step of judgment is carried out.

6. The missed detection method for road surface disease target detection according to claim 5, characterized in that: The next step of determination is specifically: Manually confirm again whether there are any missed detections. If not, re-judge the pavement defect manually annotated dataset image as having no missed detections. If so, continue to determine whether it is necessary to draw bounding boxes and add labels to the missed defects. If necessary, use the target detection and annotation tool to add additional markings. If not, do not perform any operation on the pavement defect manually annotated dataset image.

Images