An automatic extraction method for asphalt pavement repair diseases

Abstract

The present invention belongs to the field of pavement disease identification and detection, and relates to an automatic extraction method for asphalt pavement repair diseases. The steps include: image contrast enhancement, constructing a data set, improving SegNet and model training, and automatic extraction of repair diseases. Effects: The line removal + MSRCR method improves the image quality and weakens the influence of intensity non-uniformity and brightness non-uniformity on the disease image recognition and detection method. The combination of the dilated convolutional layer and the 1×1 convolutional layer effectively increases the receptive field range, can obtain more detailed feature information, and improves the problem of accuracy decline caused by information loss due to convolution and pooling in the original network. The false positives in the segmentation results are removed by using the thresholds (50 < L < 75, 2500 < A < 5000) obtained from the experiments to obtain a more accurate segmentation result.

Description

Technical Field

[0001] The present invention belongs to the field of pavement disease identification and detection in the highway industry of the transportation field, and particularly relates to a new automatic extraction method for asphalt pavement repair diseases. Background Technique

[0002] As of the end of 2020, the total mileage of highways in China has reached 5.1981 million kilometers. Highway pavement disease detection is one of the important steps to maintain road stability and is also one of the main tasks of highway management departments. The automatic identification and detection method of pavement diseases has always been a research hotspot in the transportation field. Traditional pavement disease extraction methods often use features such as the gray scale, texture, and shape of disease images, and realize the extraction of asphalt pavement diseases through manually designed feature detectors, such as algorithms like adaptive threshold segmentation method, edge detection method, morphological method, wavelet analysis, etc. However, these algorithms have the following problems: 1) It is difficult to cope with rich and diverse pavement disease scenarios; 2) Uneven illumination leads to low recognition accuracy; 3) There are many algorithm parameters and manual intervention is required, resulting in low efficiency; 4) The resolution of a single image of the vehicle lane pavement disease is high and the amount of image processing data is large, and the above methods are difficult to achieve rapid and accurate recognition of pavement diseases.

[0003] In recent years, thanks to the powerful non-linear fitting ability and efficient image processing ability of deep learning, methods for pavement disease recognition, classification, segmentation, detection, etc. based on deep learning have greatly improved the disease detection efficiency. However, there are still the following problems: 1) The overall quality of the image is poor due to intensity non-uniformity and brightness non-uniformity; 2) The complex pavement texture varies greatly from the texture of the intact pavement at different scales, and most segmentation methods are difficult to obtain good segmentation results. As one of the most common diseases, there is little research on pavement repair diseases in the existing technology. At the same time, the existing semantic segmentation methods have poor segmentation effects on them. Summary of the Invention

[0004] The purpose of the present invention is to provide a new automatic extraction method for asphalt pavement repair diseases, so as to solve the foregoing problems existing in the prior art.

[0005] The technical solution of the present invention: An automatic extraction method for asphalt pavement repair diseases includes the following steps:

[0006] 1) Image contrast enhancement

[0007] S1. Collect asphalt pavement repair disease images by a road detection vehicle. The number and resolution of the images meet the minimum requirements for data construction. Utilize the characteristics that the white road markings have regular shapes and the overall pixel values are much larger than the surrounding pavement, and take the average pixel values of three or four points (not within the markings) near the white road markings to fill the markings one by one to achieve the effect of removing the markings;

[0008] Calculation method for removing white road markings: Read the pixel value V at the pixel point P(i, j), P(i，j) where i and j are the row coordinate and column coordinate respectively;

[0009] If V P(i，j) > 125, then process it in the following way:

[0010] ① When j < 100, V P(i，j) is equal to the average value of P(i, j + 100), P(i - 100, j + 100), and P(i + 100, j + 100);

[0011] ② When 100 < j < 1700, V P(i，j) is equal to the average value of P(i, j - 100), P(i - 100, j - 100), P(i, j + 200), and P(i + 200, j + 200);

[0012] ③ When j > 1700, V P(i，j) is equal to the average value of P(i, j - 100), P(i - 100, j - 100), and P(i + 100, j - 100);

[0013] If V P(i，j) is less than or equal to 125, then the pixel value V at the pixel point P(i, j) P(i，j) remains unchanged.

[0014] Enhance the contrast of the obtained image with the white road markings removed through contrast enhancement software.

[0015] 2) Construct a dataset

[0016] S2. Make sample labels through data annotation software to obtain images and corresponding labels, and assign the label for patching diseases as (128, 0, 0);

[0017] S3. Crop the processed images in S1 to obtain cropped images with a cropping resolution of 256×256 pixels;

[0018] S4. Rotate the images obtained in S3 by 90°, 180°, flip them vertically and horizontally to obtain images whose quantity is a multiple of the quantity of the images in S3, and construct a training set, a test set, and a validation set according to 6:3:1.

[0019] 3) Improve SegNet and model training

[0020] S5. Based on SegNet, ResNet50 is used as the encoder. The image resolution of the network input layer is 256×256×3, representing width, length, and the number of channels respectively. Through a pooling layer and 49 convolutional layers for downsampling, the width, height, and the number of convolutional kernels of the feature map finally output by the encoder network are 8×8×2048;

[0021] S6. During the decoding process, convolutional layers with dilation rates of 4, 8, 12, 16 and a convolutional kernel of 3×3 are used to perform dilated convolution on the feature maps of 16×16×2, 32×32×2, 64×64×2, and 128×128×2 respectively, and the feature maps of D1: 8×8×2, D2: 16×16×2, D3: 32×32×2, and D4: 64×64×2 are obtained in sequence; a convolutional layer with a convolutional kernel of 1×1 is used to perform convolution on the feature map of 8×8×2, fuse it with D1 and perform upsampling by a factor of two to obtain E1: 16×16×2; a convolutional layer with a convolutional kernel of 1×1 is used to perform convolution on E1, fuse the obtained feature map with D2 and perform upsampling by a factor of two to obtain E2: 32×32×2; a convolutional layer with a convolutional kernel of 1×1 is used to perform convolution on E2, fuse the obtained feature map with D3 and perform upsampling by a factor of two to obtain E3: 64×64×2; a convolutional layer with a convolutional kernel of 1×1 is used to perform convolution on E3, fuse the obtained feature map with D4 and perform upsampling by a factor of two to obtain E4: 128×128×2; E4 is upsampled by a factor of two to obtain a feature map with the same size as the input image, and through the Softmax layer, the resolution of the feature map is generated as 256×256×2, representing its width, height, and the number of categories;

[0022] S7. Use MIoU and F1 as evaluation metrics, use cross-entropy as the loss function, and use the adaptive moment estimation optimization algorithm as the optimizer to train the network;

[0023] S8. Use the trained model to predict the pavement repair images in a certain area, select the repair images with poor prediction results (prediction accuracy less than 90%), manually label their diseases, and put the labeled labels and images into the training set to retrain the model, gradually improving the prediction accuracy of the model.

[0024] 4) Automatic extraction of repair diseases

[0025] S9. Use the established dataset to train the improved network, apply the trained model to automatically extract repair diseases, and generate a crack repair disease image segmentation dataset. The segmentation dataset includes repairs and false positives;

[0026] S10. Conduct statistical analysis on the segmentation results in S9 and conduct experiments to obtain the simultaneous use of area A and the absolute values of length and width LAs a threshold, where 50 < L <75, 2500 < A <5000, the false positives in the results can be eliminated, and a more accurate segmentation result of the repaired disease can be obtained.

[0027] Preferably, the number and resolution of the images of the asphalt pavement repair diseases collected in S1 are 890 images and 1688×1874.

[0028] Preferably, the method for enhancing the image contrast by removing the white road markings in S1 is the MSRCR method.

[0029] Preferably, the data annotation software for S2 is Labelme.

[0030] The beneficial effects of the present invention: In the above method, in S1, the marking removal + MSRCR method can improve the image quality and weaken the influence of intensity non-uniformity and brightness non-uniformity on the disease image recognition and detection method.

[0031] In S6, four dilated convolutional layers with dilation rates of 4, 8, 12, and 16 perform convolution on feature maps of different scales to obtain multi-scale feature maps; a convolutional layer with a kernel size of 1×1 performs convolution on the feature maps obtained by bilinear upsampling by a factor of 2 after each step of feature fusion. By using the combination of dilated convolutional layers and 1×1 convolutional layers, the receptive field range is effectively increased, more detailed feature information can be obtained, and the problem of accuracy degradation caused by information loss due to convolution and pooling in the original network is improved.

[0032] In S10, the experimentally obtained thresholds (50 < L < 75, 2500 < A < 5000) are used to eliminate the false positives in the segmentation results, and a more accurate segmentation result is obtained, completing the automatic extraction of asphalt pavement repair diseases. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is the flowchart of the steps of the present invention;

[0034] Figure 2 is the comparison diagram of the image enhancement processing effects;

[0035] Figure 3 is the result diagram of the sample amplification in S4;

[0036] Figure 4 is the structural diagram of the improved encoder-decoder network;

[0037] Figure 5 is the schematic diagram of the segmentation result. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0038] An automatic extraction method for asphalt pavement repair diseases, including the steps of: enhancing image contrast, constructing a dataset, improving SegNet and model training, and automatically extracting repair diseases.

[0039] 1) Enhancing image contrast

[0040] S1. Collect asphalt pavement repair disease images by a road detection vehicle. The number of images is 890 and the resolution is 1688×1874. Using the characteristics that the white road markings have regular shapes and the overall pixel values are much larger than those of the surrounding road surfaces, take the average pixel values of three or four points (not within the markings) near the white road markings and fill the markings one by one to achieve the effect of removing the markings;

[0041] Calculation method for removing white road markings: Read the pixel value V at pixel point P(i, j) P(i，j) , where i and j are the row coordinate and column coordinate respectively;

[0042] If V P(i，j) > 125, then process it in the following way:

[0043] ① When j < 100, V P(i，j) is equal to the average value of P(i, j + 100), P(i - 100, j + 100), and P(i + 100, j + 100);

[0044] ② When 100 < j < 1700, V P(i，j) is equal to the average value of P(i, j - 100), P(i - 100, j - 100), P(i, j + 200), and P(i + 200, j + 200);

[0045] ③ When j > 1700, V P(i，j) is equal to the average value of P(i, j - 100), P(i - 100, j - 100), and P(i + 100, j - 100);

[0046] If V P(i，j) is less than or equal to 125, then the pixel value V at pixel point P(i, j) P(i，j) remains unchanged.

[0047] Enhance the image contrast of the obtained image without white road markings using the MSRCR method. As Figure 2 shown in the comparison of the before and after effects of the image enhancement processing, the white in the marked area of the c diagram has been removed and the contrast has been improved.

[0048] 2) Constructing a dataset

[0049] S2. Use Labelme to make sample labels, obtain images and corresponding labels, and assign the label of the repair disease as (128, 0, 0);

[0050] S3. Crop the processed images in S1 to obtain 3,000 images with a resolution of 256×256;

[0051] S4. Rotate the images obtained in S3 by 90°, 180°, flip them vertically and horizontally to get 15,000 images, and construct a training set, a test set and a validation set according to the ratio of 6:3:1. The sample image augmentation is shown as Figure 3 shown.

[0052] 3) Improve SegNet and model training

[0053] S5. Based on SegNet, use ResNet50 as the encoder. The image resolution of the network input layer is 256×256×3, which represent the width, length and number of channels respectively. Through a pooling layer and 49 convolutional layers for downsampling, the width, height and number of convolutional kernels of the feature map finally output by the encoder network are 8×8×2048;

[0054] S6. During the decoding process, use convolutional layers with dilation rates of 4, 8, 12, 16 and a convolutional kernel of 3×3 to perform dilated convolution on the feature maps of 16×16×2, 32×32×2, 64×64×2, 128×128×2 respectively, and obtain feature maps of D1:8×8×2, D2:16×16×2, D3:32×32×2, D4:64×64×2 in turn; use a convolutional layer with a convolutional kernel of 1×1 to perform convolution on the feature map of 8×8×2, fuse it with D1 and perform two - fold upsampling to get E1: 16×16×2; use a convolutional layer with a convolutional kernel of 1×1 to perform convolution on E1, fuse the obtained feature map with D2 and perform two - fold upsampling to get E2: 32×32×2; use a convolutional layer with a convolutional kernel of 1×1 to perform convolution on E2, fuse the obtained feature map with D3 and perform two - fold upsampling to get E3: 64×64×2; use a convolutional layer with a convolutional kernel of 1×1 to perform convolution on E3, fuse the obtained feature map with D4 and perform two - fold upsampling to get E4: 128×128×2; perform two - fold upsampling on E4 to get a feature map with the same size as the input image, and generate a feature map with a resolution of 256×256×2 through the Softmax layer, which represents its width, height and number of classes;

[0055] S7. Use MIoU and F1 as evaluation metrics, use cross - entropy as the loss function, and use the adaptive moment estimation optimization algorithm as the optimizer to train the network;

[0056] S8. Use the trained model to predict the pavement repair images in a certain area, select the repair images with poor prediction results (prediction accuracy less than 90%), manually label their diseases, and put the labeled tags and images into the training set to retrain the model, gradually improving the prediction accuracy of the model.

[0057] 4) Automatic extraction of repair diseases

[0058] S9. Train the improved network with the established data set, use the trained model to automatically extract repair diseases, and generate a crack repair disease image segmentation data set. The segmentation data set includes repairs and false positives.

[0059] S10. Conduct statistical analysis on the segmentation results in S9 (including repairs and false positives), and conduct experiments to obtain the simultaneous use of area A and the absolute values of length and width L as thresholds, where 50 < L < 75, 2500 < A < 5000, then the false positives in the results can be eliminated to obtain a more accurate segmentation result of repair diseases.

Claims

1. An automatic extraction method for asphalt pavement repair diseases, characterized in that The steps include: 1) Image contrast enhancement S1. The asphalt pavement repair disease images are collected by a road detection vehicle. The number and resolution of the images meet the minimum requirements for data construction. Using the characteristics that the white road markings have regular shapes and the overall pixel values are much larger than those of the surrounding road surfaces, the average pixel values of three or four points near the white road markings are taken to fill the markings one by one to achieve the effect of removing the markings; Calculation method for removing white road markings: Read the pixel value V at the pixel point P(i, j) P(i，j) , where i and j are the row coordinate and column coordinate respectively; If V P(i，j) > 125, then process as follows: ① When j < 100, V P(i，j) is equal to the average value of P(i, j + 100), P(i - 100, j + 100), and P(i + 100, j + 100); ② When 100 < j < 1700, V P(i，j) is equal to the average of P(i, j - 100), P(i - 100, j - 100), P(i, j + 200), and P(i + 200, j + 200); ③ When j > 1700, V P(i，j) is equal to the average of P(i, j - 100), P(i - 100, j - 100) and P(i + 100, j - 100); If V P(i，j) is less than or equal to 125, then the pixel value V at pixel point P(i, j) P(i，j) remains unchanged The image obtained after removing the white road markings is enhanced in image contrast through contrast enhancement software; 2) Dataset construction S2. Sample labels are made through data annotation software to obtain images and corresponding labels. The label assignment for repair diseases is (128, 0, 0); S3. The images processed in S1 are cropped to obtain cropped images with a resolution of 256×256 pixels; S4. The cropped images obtained in S3 are rotated 90°, rotated 180°, flipped vertically, and flipped horizontally to obtain images whose quantity is a multiple of the number of images in S3. The training set, test set, and validation set are constructed according to 6:3:1; 3) Improvement of SegNet and model training S5. Based on SegNet, ResNet50 is used as the encoder. The image resolution of the network input layer is 256×256×3, representing width, length, and number of channels respectively. Through a pooling layer and 49 convolutional layers for downsampling, the width, height, and number of convolutional kernels of the feature map finally output by the encoder network are 8×8×2048; S6. During the decoding process, convolutional layers with dilation rates of 4, 8, 12, 16 and a convolutional kernel of 3×3 are used to perform dilated convolutions on the feature maps of 16×16×2, 32×32×2, 64×64×2, 128×128×2 respectively, and the feature maps of D1:8×8×2, D2:16×16×2, D3:32×32×2, D4:64×64×2 are obtained in sequence; A convolutional layer with a convolutional kernel of 1×1 is used to perform convolution on the feature map of 8×8×2, fuse it with D1 and perform two - fold upsampling to obtain E1: 16×16×2; A convolutional layer with a convolutional kernel of 1×1 is used to perform convolution on E1, fuse the obtained feature map with D2 and perform two - fold upsampling to obtain E2: 32×32×2; A convolutional layer with a convolutional kernel of 1×1 is used to perform convolution on E2, fuse the obtained feature map with D3 and perform two - fold upsampling to obtain E3:64×64×2; A convolutional layer with a convolutional kernel of 1×1 is used to perform convolution on E3, fuse the obtained feature map with D4 and perform two - fold upsampling to obtain E4: 128×128×2; E4 is upsampled by a factor of two to obtain a feature map with the same size as the input image. Through the Softmax layer, the resolution of the generated feature map is 256×256×2, representing its width, height, and number of classes; S7. MIoU and F1 are used as evaluation metrics, cross - entropy is used as the loss function, and the adaptive moment estimation optimization algorithm is used as the optimizer to train the network; S8. Use the trained model to predict the pavement repair images in a certain area, select the repair images with poor prediction results, manually label their diseases, and put the labeled tags and images into the training set to retrain the model, gradually improving the prediction accuracy of the model; 4) Automatic extraction of repair diseases S9. Train the improved network with the established data set, use the trained model to automatically extract repair diseases, generate a crack repair disease image segmentation data set, and the segmentation data set includes repairs and false positives; S10. Statistically analyze the segmentation results in S9 and conduct experiments to obtain the use of both area A and the absolute values of length and width L as thresholds, where 50 < L < 75, 2500 < A < 5000, so that false positives in the results can be eliminated to obtain a more accurate segmentation result for patching diseases.

2. The automatic extraction method for asphalt pavement repair diseases according to claim 1, characterized in that: S1 The number and resolution of the asphalt pavement repair disease images collected are 890 images and 1688×1874.

3. The automatic extraction method for asphalt pavement repair diseases according to claim 1, characterized in that: The method for enhancing the image contrast by removing the white road markings in S1 is the MSRCR method.

4. The automatic extraction method for asphalt pavement repair diseases according to claim 1, characterized in that: The data annotation software for S2 is selected as Labelme.

Images