False detection method for pavement disease detection

Abstract

The invention relates to a pavement disease detection-oriented false detection method, and belongs to the field of quality improvement research of highway pavement disease detection. According to the method, manually labeled data sets of different pavement diseases are collected, the number of disease images is expanded through a'disease expansion 'image processing technology to make a data set for training a false detection model, the false detection model is trained by using the data set, and then the classification effect of the model is checked. By comparing a model classification result with an original label type, whether a false detection problem exists or not can be judged, and a corresponding label data line in a label file corresponding to the image can be accurately positioned for disease type correction. According to the method, the false detection model capable of automatically distinguishing different pavement disease types is trained, the label data position needing to be modified can be positioned, and the false detection condition of the pavement disease target detection result image can be improved.

Description

Technical Field

[0001] The invention relates to a false detection method for road surface disease detection, and belongs to the research field of quality improvement of highway road surface disease detection. Background Art

[0002] Asphalt pavement is the most widely used high-grade pavement in road construction. With the development of China's self-produced road asphalt material industry, asphalt pavement has been widely used in urban roads and highway trunk lines, becoming the most paved high-grade pavement in China. However, with the increase in traffic volume, frequent weather changes and other factors, various pavement diseases continue to appear, and the types of pavement diseases include but are not limited to: cracks, repairs and potholes. The work of the present invention involves six different pavement diseases, namely cracks, cracks, looseness, potholes, oil spills and repairs. The existing target detection false detection inspection work is mainly completed by manual inspection. However, long-term manual inspection work may cause the problem of false detection of diseases due to factors such as insufficient energy or decreased attention of the labeling personnel. At present, there is no good way to automatically solve the false detection problem of pavement disease target detection. The existing false detection work can only rely on manual review of the target detection results one by one, that is, to check whether there is a false detection of the disease type within the bounding box, and whether there is a problem of missing and undetected diseases outside the bounding box. After multiple reviews and corrections, a more accurate target detection result can be finally obtained. This method is inefficient, consumes a lot of human resources, and the annotation levels of different reviewers are inconsistent and subjective, which may lead to inconsistent results after review. Summary of the invention

[0003] The purpose of the present invention is to provide a false detection method for road surface disease detection, aiming to solve the technical problem that the existing false detection work can only rely on manual review of target detection results one by one, which is difficult to ensure the false detection efficiency and accuracy.

[0004] The technical solution of the present invention is: the present invention designs a data set preprocessing method called "disease expansion" image processing technology, and uses this method to unify the hand-labeled data set of road surface diseases to make a data set of false detection detection model, so as to train a false detection model (multi-classification model) based on the data set. The model can determine which category of different diseases the content in the disease expansion image processed by any bounding box in the false label data set belongs to, and then by comparing with the original label type, it can be determined whether there is a false detection problem. This method can be used for false detection detection of target detection result images.

[0005] A false detection method for pavement disease detection, the specific steps are: Step 1: Collect manually annotated datasets of different pavement defects; Step 2: Use the “disease augmentation” image processing technology to process the manually annotated data set into a false positive data set containing different diseases; Step 3: Based on the classification model, use the false detection data set for training to obtain the false detection model; Step 4: Use the false positive model to output the classification result type of the false positive data set; Step 5: Compare the classification result type and the original image label type to determine whether there is a false detection problem, and locate and correct the image with the false detection problem.

[0006] The Step 1 is specifically as follows: Step 1.1: Obtain a manually labeled dataset of road surface defects. All images are grayscale images collected by the inspection vehicle and are stored in image subfolders under different folders. Step 1.2: According to the manually labeled data set, find the label text files corresponding to the image names one by one and store them in the label folders of different diseases; Step 1.3: Use annotation software to check and correct the manually annotated dataset.

[0007] The Step 2 is specifically as follows: Step 2.1: Obtain a manually labeled dataset containing any number of pavement disease bounding boxes; Step 2.2: According to the Yolo format label corresponding to the image, obtain the label data corresponding to the unprocessed bounding box in order in rows, and calculate the basic parameters of the corresponding bounding box according to the label data; Step 2.3: Crop the corresponding bounding box according to the calculated basic parameters and name it as the disease frame to crop the image; Step 2.4: Crop the long side of the defect frame image and resize it to X pixels in length, and resize the short side and the long side in proportion to make the image aspect ratio consistent; Step 2.5: Calculate the height and width ratio of the cropped image of the defect frame. If the ratio of the long side to the short side is between [1, 2], no processing is performed; otherwise, resize the short side to X / 2 pixels or resize the long side to X pixels. Step 2.6: Sharpen the image cropped from the diseased frame and calculate the grayscale average; Step 2.7: Prepare a new normal road surface image without any disease as the background image, adjust the gray value of the background image according to the obtained gray value average, and then resize it to X*X pixel value; Step 2.8: Replace the background image pixels point by point with the damaged frame to crop the image and obtain the damaged adjusted image; Step 2.9: Perform horizontal, vertical, and horizontal-vertical flip operations on the diseased adjusted image to obtain the feature and original Figure 1 The same mirror-flipped image is then stitched together into an enlarged image with a height and width twice that of the disease-adjusted image and an area four times that of the disease-adjusted image, which is named the disease-enlarged image. Step 2.10: Rename the file of the disease expansion image; Step 2.11: According to the disease type of the original bounding box label corresponding to the disease expansion image, store it in the corresponding disease folder as a data set sample of the false detection model; Step 2.12: Return to Step 2.2 and obtain the label data rows corresponding to all bounding boxes of the image; if all bounding boxes in the image have been processed, return to Step 2.1 to obtain a new manually annotated dataset; if all images in the manually annotated dataset have been processed, the false detection dataset of the false detection model is completed.

[0008] The Step 5 is specifically as follows: Compare the classification result type of the model with the label type of the original image. If the disease types are consistent, it is determined that the label data of the bounding box corresponding to the image has no problem of misdetecting the disease type; otherwise, it is determined that the label data of the bounding box corresponding to the image may have a problem of misdetecting the disease type, and the next step of determination is performed.

[0009] The next step of determination is specifically: Manually confirm whether there is a false detection problem again. If not, re-judge the image as having no missed defects. If so, locate the label data that needs to be modified in the nth line by checking the last character n in the image file name, and then modify the label data in the nth line to the correct disease type to complete the correction of the false detection problem.

[0010] The beneficial effects of the present invention are: The present invention has produced a false positive detection data set based on the "disease expansion" image processing technology. The data set includes six different disease types, and the false positive detection model based on the classification model is trained using the data set. The false positive detection model can automatically and batch predict which disease type the false positive detection data set image belongs to, and finally compare the model classification result with the original disease type annotated on the image. If the two are inconsistent, it is considered that there may be a false positive detection problem, and it is necessary to further return to the original label data for modification. The use of this invention can automatically predict the disease type of the false positive detection data set through the model, thereby reducing time and human resources. At the same time, because it has a high classification accuracy, the reliability of false positive detection judgment is also relatively good. Compared with the previous cumbersome pure manual review of false positive detection work, the present invention not only improves efficiency, and has a higher and more stable detection accuracy, but also can reduce labor and time costs at the same time, providing a more efficient means for false positive detection of disease target detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a flow chart of the steps of the present invention; Figure 2 It is a simplified diagram of the disease expansion method of the present invention; Figure 3 It is a partial illustration of the disease expansion method of the present invention (I); Figure 4 It is a partial illustration of the disease expansion method of the present invention (II); Figure 5 It is a diagram showing the classification prediction results of the false detection model of the present invention in two test sets; Figure 6 is a confusion matrix classification result diagram of the false detection model of the present invention on the test set 1; Figure 7 It is a confusion matrix classification result diagram of the false detection model of the present invention on the test set 2; Figure 8 This is a display diagram of the classification results of the model of the present invention (I); Fig. 9 This is the display diagram of the classification results of the model of the present invention (II); Fig.10 This is the display diagram of the classification results of the model of the present invention (III); Fig.11 is a flow chart of the false detection judgment method and label positioning method of the present invention (I); Fig.12 This is a flow chart of the false detection judgment method and label positioning method of the present invention (II). DETAILED DESCRIPTION

[0012] The present invention will be further described below in conjunction with the accompanying drawings and specific implementation methods.

[0013] Example 1: Figure 1 As shown in FIG. 1 , a false detection method for pavement disease detection is provided, and the specific steps are as follows: Step 1: Collect manually annotated datasets of different pavement defects.

[0014] Step 1.1: Obtain a manually labeled dataset of road surface defects. All images are grayscale images collected by the inspection vehicle and are stored in image subfolders under different folders. Specifically, manually annotated datasets of six different types of defects were obtained, including cracks, fissures, pits, looseness, oil spills, and repairs. The number of images of each type of defect was consistent with 40, which were stored in the image subfolders of the six different defect folders (the folder names of the six defects were LIE_FENG, JUN_LIE, FAN_YOU, KENG_CAO, XIU_BU, and SONG_SAN, respectively).

[0015] Step 1.2: According to the manually annotated data set, find the label text files corresponding to the image names one by one and store them in the label subfolders of the six different disease folders; Step 1.3: Use labelimg software to check and correct all disease dataset images to ensure that there are no missed labels or false detections.

[0016] Step 2: Use the “disease augmentation” image processing technology to process the manually annotated dataset into a false positive dataset containing different diseases.

[0017] Specifically, we used the “disease augmentation” image processing technique to process all manually annotated datasets of six different diseases and produced a simplified diagram of the false positive dataset. Figure 2 The disease expansion method is an image processing method that uses the Python programming language and imports the cv2 module to perform image processing, and can realize automatic and batch generation of false positive model data sets. Further: Step 2.1: Obtain a manually labeled dataset containing any number of pavement disease bounding boxes; Step 2.2: According to the Yolo format label corresponding to the image, obtain the label data corresponding to the unprocessed bounding box in order in rows, and calculate the basic parameters such as the center point position, height and width of the corresponding bounding box according to the label data; Step 2.3: Crop the corresponding bounding box according to the calculated basic parameters and name it as the disease frame to crop the image; Step 2.4: The long side of the defect frame is resized to 300 pixels, and the short side is resized proportionally with the long side to make the image aspect ratio consistent; Step 2.5: Calculate the height and width ratio of the cropped image of the defect frame. If the ratio of the longer side to the shorter side is between [1, 2], no processing is performed. Otherwise, if the shorter side is too short, the shorter side needs to be resized to 150 pixels. Otherwise, if the longer side is too long, it needs to be resized to 300 pixels. Step 2.6: Sharpen the image cropped from the diseased frame and calculate the grayscale average; Step 2.7: Prepare a new normal road surface image without any disease as the background image, adjust the gray value of the background image according to the obtained gray value average, and then resize it to 300*300 pixels; Step 2.8: Replace the background image pixels point by point with the damaged frame to crop the image and obtain the damaged adjusted image; Step 2.9: Flip the diseased image horizontally, vertically, and horizontally and vertically to obtain three other feature and original images. Figure 1 Finally, these four images are stitched together into an enlarged image with a height and width twice that of the disease adjustment image and an area four times that of the disease adjustment image, named the disease expansion image. Some disease expansion methods are shown in the figure. Figure 3 and Figure 4 As shown; Step 2.10: Rename the file of the disease expansion image in the following format (taking crack disease as an example): L_082717381-002174-002174-1_1. Among them, 'L' corresponds to the crack type; '082717381-002174-002174-1' corresponds to the image name to which the disease belongs; the number '1' at the end indicates the first row of the label data of the bounding box corresponding to the disease, and if the number is '2' or '3', it indicates the second or third row of the label data of the bounding box corresponding to the disease; Step 2.11: According to the disease type of the original bounding box label corresponding to the disease expansion image, store it in the corresponding disease folder (the false detection data set includes six disease classification folders, namely L, J, K, S, F, and X) as the data set sample of the false detection model; Step 2.12: Return to Step 2.2 and obtain the label data rows corresponding to all bounding boxes of the image; if all bounding boxes in the image have been processed, return to Step 2.1 to obtain a new manually annotated dataset; if all images in the manually annotated dataset have been processed, the false detection dataset of the false detection model is completed.

[0018] Step 3: Based on the classification model, use the false detection data set for training to obtain the false detection model.

[0019] Step 3.1: Download the official yolov5 version 7.0 from the github website. This version contains the yolov5s-cls.pt file that can be used for classification tasks. The classification task of the present invention can be performed based on this model.

[0020] Step 3.2: Divide the obtained false detection model data set into training, validation, and test sets, with the ratio of the three being 8:1:1; then rotate the test set 90° clockwise, and finally the ratio of the three is 8:1:2.

[0021] Step 3.3: Train the false detection model for the six-category task. The main hyperparameter settings are shown in Table 1: Table 1 Hyperparameter settings

[0022] Step 3.4: After training, the false detection model is obtained, which can automatically and batch-wise infer the disease type of the disease-expanded image, and provide the disease classification results of the false detection model for subsequent comparison with the original label type.

[0023] Step 4: Use the false positive model to output the classification result type of the false positive data set.

[0024] Specifically, the trained false positive model is used to predict the disease classification of the test set. The predict.py file is run, and the model classifies and predicts each disease expansion image in the test set to infer its disease type. After the prediction is completed, the classification results are saved in the same folder. The present invention has designed two test sets in total, and achieved high classification accuracy rates of 91% and 86% respectively. The results of the two tests are shown in Figure 2. Figure 5 shown.

[0025] The confusion matrix classification results of the test set false detection model in test sets one and two are as follows Figure 6 and Figure 7 shown.

[0026] Step 5: Compare the classification result type and the original image label type to determine whether there is a false detection problem, and locate and correct the image with the false detection problem.

[0027] Step 5.1: Check the model classification results. Each result image is in the form of "disease expansion image + disease classification probability number in the upper left corner". The capital letter corresponding to the maximum probability in the first row in the upper left corner represents one of the six different disease types, that is, the disease type that the model infers that the disease contained in the image is most likely to belong to. The model classification result example is shown in the figure below: Figure 8 , 9 , as shown in 10.

[0028] Step 5.2: Compare the capital letters corresponding to the first row of probabilities (i.e., the maximum probability) in the upper left corner of the image. If the letter is consistent with the first letter of the image file name, it is determined that the label of the bounding box corresponding to the disease has no problem of misdetecting the disease type; otherwise, if the letters are inconsistent, it is determined that the label data of the bounding box corresponding to the image may have a problem of misdetecting the disease type, and the next step of determination is performed.

[0029] The next step of determination is specifically: Manually confirm whether there is a false detection problem again. If not, re-judge the image as having no missed defects. If so, locate the label data that needs to be modified in the nth line by checking the last character n in the image file name, and then modify the label data in the nth line to the correct disease type to complete the correction of the false detection problem.

[0030] The flowchart of the false detection judgment method and label positioning method is as follows Fig.11 and Fig.12 shown.

[0031] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A false detection method for road surface disease detection, characterized in that: The specific steps of the method are: Step 1: Collect manually annotated datasets of different pavement defects; Step 2: Use the "disease augmentation" image processing technology to process the manually annotated data set into a false positive data set containing different diseases; Step 3: Based on the classification model, use the false detection data set for training to obtain the false detection model; Step 4: Use the false positive model to output the classification result type of the false positive data set; Step 5: Compare the classification result type and the original image label type to determine whether there is a false detection problem, and locate and correct the image with the false detection problem.

2. The false detection method for pavement disease detection according to claim 1, characterized in that: The Step 1 is specifically as follows: Step 1.1: Obtain a manually labeled dataset of road surface defects. All images are grayscale images collected by the inspection vehicle and are stored in image subfolders under different folders. Step 1.2: According to the manually labeled data set, find the label text files corresponding to the image names one by one and store them in the label folders of different diseases; Step 1.3: Use annotation software to check and correct the manually annotated dataset.

3. The false detection method for pavement damage detection according to claim 1, characterized in that: The Step 2 is specifically as follows: Step 2.1: Obtain a manually labeled dataset containing any number of pavement disease bounding boxes; Step 2.2: According to the Yolo format label corresponding to the image, obtain the label data corresponding to the unprocessed bounding box in order in rows, and calculate the basic parameters of the corresponding bounding box according to the label data; Step 2.3: Crop the corresponding bounding box according to the calculated basic parameters and name it as the disease frame to crop the image; Step 2.4: Crop the long side of the defect frame image and resize it to X pixels in length, and resize the short side and the long side in proportion to make the image aspect ratio consistent; Step 2.5: Calculate the height and width ratio of the cropped image of the defect frame. If the ratio of the long side to the short side is between [1, 2], no processing is performed; otherwise, resize the short side to X / 2 pixels or resize the long side to X pixels. Step 2.6: Sharpen the image cropped from the diseased frame and calculate the grayscale average; Step 2.7: Prepare a new normal road surface image without any disease as the background image, adjust the gray value of the background image according to the obtained gray value average, and then resize it to X*X pixel value; Step 2.8: Replace the background image pixels point by point with the damaged frame to crop the image and obtain the damaged adjusted image; Step 2.9: Flip the disease-adjusted image horizontally, vertically, and horizontally and vertically to obtain a mirror-flipped image with the same features as the original image. Finally, splice the images into an enlarged image with a height and width twice that of the disease-adjusted image and an area four times that of the disease-adjusted image, named the disease-enlarged image; Step 2.10: Rename the file of the disease expansion image; Step 2.11: According to the disease type of the original bounding box label corresponding to the disease expansion image, store it in the corresponding disease folder as a data set sample for the false detection model; Step 2.12: Return to Step 2.2 and obtain the label data rows corresponding to all bounding boxes of the image; if all bounding boxes in the image have been processed, return to Step 2.1 to obtain a new manually annotated dataset; if all images in the manually annotated dataset have been processed, the false detection dataset of the false detection model is completed.

4. The false detection method for pavement disease detection according to claim 1, characterized in that: The Step 5 is specifically as follows: Compare the classification result type of the model with the label type of the original image. If the disease types are consistent, it is determined that the label data of the bounding box corresponding to the image has no problem of misdetecting the disease type; otherwise, it is determined that the label data of the bounding box corresponding to the image may have a problem of misdetecting the disease type, and the next step of determination is performed.

5. The false detection method for pavement disease detection according to claim 4, characterized in that: The next step of determination is specifically: Manually confirm whether there is a false detection problem again. If not, re-judge the image as having no missed defects. If so, locate the label data that needs to be modified in the nth line by checking the last character n in the image file name, and then modify the label data in the nth line to the correct disease type to complete the correction of the false detection problem.

Images