A false detection method for road surface disease detection

Abstract

The present application relates to a kind of false detection methods for road surface disease detection, belong to the quality promotion research field of highway pavement disease detection.The present application collects the data set of different pavement disease manual annotation, expands the number of disease image by "disease expansion" image processing technology to make the data set of training false detection model, and uses data set to train false detection model, and then verifies the classification effect of model.By comparing model classification result and original label type, whether it exists false detection problem can be judged, and corresponding label data line in image corresponding label file can be accurately positioned to carry out disease type correction.The present application trains the false detection model capable of automatically distinguishing different pavement disease types, can locate the label data position needing modification, and can improve the false detection situation of pavement disease target detection result image.

Description

TECHNICAL FIELD

[0001] The present application relates to a mis-detection method for road surface disease detection, belonging to the field of quality improvement of road surface disease detection. BACKGROUND

[0002] Asphalt pavement is a kind of high-grade pavement most widely used in road construction. With the development of China's self-produced road asphalt material industry, asphalt pavement has been widely used in urban roads and highway trunk lines, and has become the most widely used high-grade pavement in China. However, with the increase of traffic flow, frequent changes in weather and other factors, various road diseases are constantly emerging, including but not limited to cracks, repairs and potholes. The work of the present application involves six different road diseases, which are cracks, cracks, loose, potholes, oil and repairs. The existing target detection mis-detection checking work is mainly completed by human checking, but long-time human checking work may cause mis-detection of diseases due to lack of energy or attention decline of the labeling personnel, etc., resulting in the problem of mis-detection of diseases. There is no good way to automatically solve the mis-detection problem of road disease target detection. The existing mis-detection checking work only relies on human rechecking of the target detection results, i.e. checking whether there is mis-detection of disease type in the bounding box and whether there is missing detection of disease outside the bounding box. After multiple reviews and corrections, more accurate target detection results can be obtained. This way is inefficient, consumes a lot of human resources, and different reviewers have different labeling levels, which is subjective and may cause inconsistent results after review. SUMMARY

[0003] The present application aims to provide a mis-detection method for road surface disease detection, which aims to solve the technical problem that the existing mis-detection checking work only relies on human rechecking of the target detection results, which is difficult to guarantee the mis-detection efficiency and accuracy.

[0004] The technical solution of the present application is: the present application designs a data set preprocessing method named "disease expansion" image processing technology, and uses the method to unify the manual data set of road surface disease into a data set for mis-detection model, so as to train a mis-detection model (multi-classification model) based on it. The model can judge which category the content in the disease expansion image processed by any bounding box in the mis-labeled data set belongs to, and then compare it with the original label type to judge whether there is mis-detection problem. This method can be used for mis-detection of target detection result images.

[0005] A mis-detection method for road surface disease detection, the specific steps are: A mis-detection method for road surface disease detection, the specific steps are:

[0006] Step1: Collecting the manually annotated dataset of different road surface diseases;

[0007] Step2: Using the "disease expansion" image processing technology, the manually annotated dataset is processed into a misclassification dataset containing different diseases;

[0008] Step3: Based on the classification model, the misclassification dataset is used for training to obtain a misclassification model;

[0009] Step4: Using the misclassification model to output the classification result type of the misclassification dataset;

[0010] Step5: Comparing the classification result type and the original image label type, making a judgment on whether there is a misclassification problem, and positioning and correcting the images with misclassification problems.

[0011] The Step1 is specifically:

[0012] Step1.1: Obtain the manually annotated dataset of road surface diseases, and the images are gray-scale images collected by the detection vehicle, and are stored in the image subfolder under different folders;

[0013] Step1.2: According to the manually annotated dataset, find the label text file corresponding to the image name one by one, and store it in the label folder of different diseases;

[0014] Step1.3: Check and correct the manually annotated dataset using the annotation software.

[0015] The Step2 is specifically:

[0016] Step2.1: Obtain the manually annotated dataset containing any number of road surface disease bounding boxes;

[0017] Step2.2: According to the yolo format label corresponding to the image, the label data corresponding to the unprocessed bounding box is obtained in order according to the line unit, and the basic parameters of the corresponding bounding box are calculated according to the label data;

[0018] Step2.3: According to the calculated basic parameters, the corresponding bounding box frame is cropped, named as disease frame cropped image;

[0019] Step2.4: Resize the long side of the disease frame cropped image to X pixels in length, and resize the short side and long side at the same time to keep the aspect ratio consistent;

[0020] Step2.5: Calculate the height and width ratio of the disease frame cropped image, if the ratio of the long side and the short side is between [1, 2], no processing is performed; otherwise, the short side is resized to X / 2 pixel length or the long side is resized to X pixel length.

[0021] Step2.6: Sharpen the disease frame cropped image, and calculate the average gray value.

[0022] Step2.7: Prepare a new normal road image without disease as a background image, adjust the gray value of the background image according to the obtained average gray value, and resize the size to X*X pixel value;

[0023] Step2.8: Replace the background image pixel by pixel with the disease frame cropped image to obtain a disease adjustment image.

[0024] Step2.9: Perform horizontal, vertical, and horizontal and vertical flip operations on the disease adjustment image to obtain mirror image flip images of the same characteristics and original images, and finally splice the images into an expanded image with a height and width of twice the disease adjustment image and an area of four times, named as a disease expansion image. Figure One

[0025] Step2.10: Rename the file of the disease expansion image.

[0026] Step2.11: According to the label disease type of the original bounding box corresponding to the disease expansion image, store it in the corresponding disease folder as a data set sample of the false detection model.

[0027] Step2.12: Return to Step2.2 to obtain the label label data line corresponding to all bounding boxes of the image; if all bounding boxes in the image are processed, return to Step2.1 to obtain a new manually labeled data set; if all images in the manually labeled data set are processed, complete the false detection data set of the false detection model.

[0028] The Step5 is specifically:

[0029] Compare the classification result type of the model with the original image label type, if the disease types are consistent, it is determined that the label data of the bounding box corresponding to the image has no false detection disease type problem; otherwise, it is determined that the label data of the bounding box corresponding to the image may have a false detection disease type problem, and the next step of determination is performed.

[0030] The next step of determination is specifically:

[0031] ​The artificial reconfirms whether there is a mis-detection problem, if not, re-determine the image as no disease missing detection condition; if there is, by looking at the last character n of the image file name to locate the label data that needs to be modified in the nth row, and then modify the nth row of label data to the correct disease type, complete the correction of mis-detection problem.

[0032] The beneficial effects of the present application are:

[0033] The present application uses the "disease expansion" image processing technology to make a mis-detection data set, which includes six different disease types, and uses the data set to train a mis-detection model based on a classification model. The mis-detection model can automatically and batch predict which disease type the mis-detection data set image belongs to, and finally compare the model classification result with the original disease type of the image label, if they are not consistent, it is considered that there may be mis-detection problem, which needs to be further modified to the original label data. Using the present application can automatically predict the disease type of the mis-detection data set by the model, thereby reducing time and human resources, and because it has high classification accuracy, the reliability of mis-detection judgment is also relatively good. Compared with the previous tedious manual review of mis-detection work, the present application not only improves the efficiency, but also has high and stable detection accuracy, and can also reduce labor and time cost, providing a more efficient means for mis-detection of disease target detection results. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 is a step flow chart of the present application;

[0035] Figure 2 is a brief diagram of the disease expansion method of the present application;

[0036] Figure 3 is a partial legend display (one) of the disease expansion method of the present application;

[0037] Figure 4 is a partial legend display (two) of the disease expansion method of the present application;

[0038] Figure 5 is a classification prediction result display diagram of the mis-detection model of the present application on two test sets;

[0039] Figure 6 is a confusion matrix classification result diagram of the mis-detection model of the present application on test set one;

[0040] Figure 7 is a confusion matrix classification result diagram of the mis-detection model of the present application on test set two;

[0041] Figure 8 is a model classification result display diagram (one) of the present application;

[0042] Figure 9is a model classification result display diagram (two) of the present application;

[0043] Figure 10 is a model classification result display diagram (three) of the present application;

[0044] Figure 11 is a flowchart of the false detection judgment method and label positioning method (one) of the present application;

[0045] Figure 12 is a flowchart of the false detection judgment method and label positioning method (two) of the present application. DETAILED DESCRIPTION

[0046] The present application will be further described below in combination with the drawings and specific embodiments.

[0047] Embodiment 1: As shown in the figure, a false detection method for road surface disease detection, the specific steps are as follows: Figure 1

[0048] Step 1: Collecting manually annotated data sets of different road surface diseases.

[0049] Step 1.1: Obtain manually annotated data sets of road surface diseases, all images are gray-scale images collected by detection vehicles, and are stored in image sub-folders under different folders;

[0050] Specifically, manually annotated data sets of six different diseases are obtained, including cracks, crazing, potholes, loose, oil, and repair, and the number of images of each disease type is consistent, 40 images, which are stored in the image sub-folders of the six different disease folders (the folder names of the six diseases are LIE_FENG, JUN_LIE, FAN_YOU, KENG_CAO, XIU_BU and SONG_SAN).

[0051] Step 1.2: According to the manually annotated data set, find the label text file corresponding to the image name one by one, and store it in the label sub-folder of the six different disease folders;

[0052] Step 1.3: Use the labelimg software to check and correct all disease data set images to ensure that there is no missing label or false detection.

[0053] Step 2: Using the "disease expansion" image processing technology, the manually annotated data set is processed into a false detection data set containing different diseases.

[0054] Specifically, all manually annotated data sets of six different diseases are processed using the "disease expansion" image processing technology, and a brief diagram of the false detection data set is as shown in Figure 2 ​The disease expansion method is an image processing method that uses the Python programming language and imports the cv2 module for image processing, which can automatically and batch generate false positive model datasets. It further:

[0055] Step 2.1: Obtain a manually labeled dataset containing any number of pavement defect bounding boxes;

[0056] Step 2.2: According to the Yolo format label corresponding to the image, obtain the label data corresponding to the unprocessed bounding box in order in rows, and calculate the basic parameters of the corresponding bounding box such as the center point position, height and width based on the label data;

[0057] Step 2.3: Crop the corresponding bounding box according to the calculated basic parameters and name it the disease frame to crop the image;

[0058] Step 2.4: Crop the image of the defect frame and resize the long side to 300 pixels. The short side and the long side are resized proportionally to make the image aspect ratio consistent.

[0059] Step 2.5: Calculate the height and width ratio of the cropped image of the defect frame. If the ratio of the longer side to the shorter side is between [1, 2], no processing is performed. Otherwise, if the shorter side is too short, resize it to 150 pixels. Conversely, if the longer side is too long, resize it to 300 pixels.

[0060] Step 2.6: Sharpen the cropped image of the diseased frame and calculate the grayscale average;

[0061] Step 2.7: Prepare a new image of a normal road surface without any damage as the background image. Adjust the grayscale value of the background image according to the obtained grayscale average value, and then resize it to 300*300 pixels.

[0062] Step 2.8: Replace the background image pixels point by point with the damaged frame to crop the image and obtain the damaged image.

[0063] Step 2.9: Flip the diseased image horizontally, vertically, and horizontally and vertically to obtain three more features and the original image. Figure One Finally, these four images are stitched together to form an enlarged image with a height and width twice that of the disease adjustment image and an area four times that of the disease adjustment image. This image is named the disease expansion image. Some disease expansion methods are shown in the figure. Figure 3 and Figure 4 As shown;

[0064] Step 2.10: Rename the file of the expanded defect image to the following format (taking crack defects as an example): L_082717381-002174-002174-1_1. Here, 'L' corresponds to the crack type; '082717381-002174-002174-1' corresponds to the image name to which the defect belongs; and the number '1' at the end indicates that the defect is in the first row of the bounding box label data. If the number is '2' or '3', it indicates the second or third row of the bounding box label data.

[0065] Step 2.11: According to the disease type of the original bounding box label corresponding to the disease expansion image, it is stored in the corresponding disease folder (the false detection dataset contains a total of six disease classification folders, namely L, J, K, S, F, and X), which serves as the dataset sample of the false detection model;

[0066] Step 2.12: Return to Step 2.2 and obtain the label data rows corresponding to all bounding boxes in the image; if all bounding boxes in the image have been processed, return to Step 2.1 to obtain a new manually annotated dataset; if all images in the manually annotated dataset have been processed, the false detection dataset of the false detection model is completed.

[0067] Step 3: Based on the classification model, use the false detection dataset for training to obtain the false detection model.

[0068] Step 3.1: Download the official yolov5 version 7.0 from the github website. This version contains the yolov5s-cls.pt file that can be used for classification tasks. The classification task of the present invention can be performed based on this model.

[0069] Step 3.2: Divide the obtained false positive model dataset into training, validation, and test sets with a ratio of 8:1:1; then rotate the test set 90° clockwise, and the final ratio of the three is 8:1:2.

[0070] Step 3.3: Train the false detection model for the six-category classification task. The main hyperparameter settings are shown in Table 1:

[0071] Table 1 Hyperparameter settings

[0072]

[0073] Step3.4: After the training, the misclassification model is obtained, which can automatically and batch infer the disease type of the disease expansion image, and provide the disease classification result of the misclassification model for the subsequent comparison with the original label type.

[0074] Step4: Use the misclassification model to output the classification result type of the misclassification data set.

[0075] Specifically, using the trained misclassification model, the test set is classified and predicted, and the predict.py file is run. The model classifies and predicts each disease expansion image of the test set to infer its disease type. After prediction, the classification result is saved to the same folder. The present application designs two test sets, which achieve a high classification accuracy (accuracy) of 91% and 86% respectively. The test results are shown in Figure 5

[0076] The confusion matrix classification results of the test set misclassification model in test set one and two are shown in Figure 6 and Figure 7 .

[0077] Step5: Compare the classification result type and the original image label type to determine whether there is a misclassification problem, and locate and correct the images with misclassification problems.

[0078] Step5.1: View the obtained model classification result, each result image is in the form of “disease expansion image + disease classification probability number in the upper left corner”. The maximum probability in the first row of the upper left corner corresponds to a capital letter representing one of the six different disease types, i.e. the disease type that the model infers the image to contain is most likely to belong to. The model classification result example is shown in Figure 8 , 9 , 10.

[0079] Step5.2: Compare the capital letter corresponding to the first row of probability in the upper left corner of the image (i.e. the maximum probability). If the letter is consistent with the first letter of the image file name, it is determined that the label of the bounding box corresponding to the disease has no misclassification disease type problem; otherwise, if the letters are not consistent, it is determined that the label data of the bounding box corresponding to the image may have a misclassification disease type problem, and the next step of determination is performed.

[0080] The next step of determination is specifically:

[0081] ​The artificial re-confirmation is whether there is a mis-detection problem, if not, the image is re-determined as no disease missing detection condition; if there is, by looking at the last character n of the image file name to locate the label data that needs to be modified in the nth row, and then modify the nth row of label data to the correct disease type, complete the correction of mis-detection problem.

[0082] The flow chart of mis-detection judgment mode and label positioning mode is shown in Figure 11 and Figure 12 .

[0083] The specific embodiments of the application are described in detail above with reference to the accompanying drawings, but the application is not limited to the above embodiments, and various changes can be made within the knowledge of those skilled in the art without departing from the purpose of the application.

Claims

1. A false detection method for road surface disease detection, characterized in that, The method comprises the following steps: Step 1: collect manually annotated data sets of different road surface diseases; Step 2: use the "disease expansion" image processing technology to process the manually annotated data set into a misjudgment data set containing different diseases; Step 3: based on the classification model, use the misjudgment data set for training to obtain a misjudgment model; Step 4: use the misjudgment model to output the classification result type of the misjudgment data set; Step 5: compare the classification result type with the original image label type to determine whether there is a misjudgment problem, and locate and correct the images with misjudgment problems; The Step 2 comprises the following steps: Step 2.1: obtain a manually annotated data set containing any number of road surface disease bounding boxes; Step 2.2: according to the yolo format label corresponding to the image, obtain the label data corresponding to the bounding box in sequence, and calculate the basic parameters of the corresponding bounding box according to the label data; Step 2.3: according to the calculated basic parameters, the corresponding bounding box frame is cut out and named as a disease frame cutting image; Step 2.4: resize the long side of the disease frame cutting image to X pixels in length, and resize the short side and long side in proportion to make the image aspect ratio consistent; Step 2.5: calculate the height and width ratio of the disease frame cutting image, if the ratio of the long side and the short side is between 1 and 2, do not process; otherwise, resize the short side to X / 2 pixels in length or resize the long side to X pixels in length; Step 2.6: sharpen the disease frame cutting image and calculate the average gray value; Step 2.7: prepare a new normal road surface image without disease as a background image, adjust the gray value of the background image according to the obtained average gray value, and resize the size to X*X pixels; Step 2.8: replace the background image pixel by pixel with the disease frame cutting image to obtain a disease adjustment image; Step 2.9: horizontally, vertically, and horizontally and vertically flip the disease adjustment image respectively to obtain mirror image flipping images with the same features as the original image, and finally splice the images into an expanded image with a height and width of twice the disease adjustment image and an area of four times, named as a disease expansion image; Step 2.10: rename the disease expansion image file; Step 2.11: according to the label disease type of the original bounding box corresponding to the disease expansion image, store it in the corresponding disease folder as a data set sample of the misjudgment model; Step 2.12: return to Step 2.2 to obtain the label data line corresponding to all bounding boxes of the image; if all the bounding boxes in the image are processed, return to Step 2.1 to obtain a new manually annotated data set; if all the images in the manually annotated data set are processed, complete the misjudgment data set of the misjudgment model.

2. The false detection method for road surface disease detection according to claim 1, wherein, The Step1 specifically is: Step1.1: Obtain the manual annotation dataset of road diseases, the images are all gray images collected by the detection vehicle, and are respectively stored in the image subfolder under different folders; Step1.2: According to the manual annotation dataset, find the label text file corresponding to the image name one by one, and respectively store them in the label folder of different diseases; Step1.3: Check and correct the manual annotation dataset using the annotation software.

3. The false detection method for road surface disease detection according to claim 1, wherein, The Step5 specifically is: Compare the classification result type of the model with the original image label type, if the disease type is consistent, it is determined that the label data of the bounding box corresponding to the image is correct, and the disease type problem is not detected; Otherwise, it is determined that the label data of the bounding box corresponding to the image may have the problem of misjudging the disease type, and the next step of determination is performed.

4. The false detection method for road surface disease detection according to claim 3, wherein, The next step of determination specifically is: Artificially confirm whether there is a misjudgment problem, if not, re-determine that the image is a no-disease missing detection case; If there is, locate the label data to be modified in the nth row by viewing the last character n of the image file name, and then modify the nth row of label data to the correct disease type to complete the correction of the misjudgment problem.

Images