Semantic segmentation method of pavement defects based on self-supervised learning

Abstract

The present invention discloses a method for semantic segmentation of pavement defects based on self-supervised learning, including a. image cropping, randomly cropping a pavement image into several sub-images; b. generating pseudo-labels, using several semantic segmentation models to perform semantic segmentation on the sub-images, generating pseudo-labels, and using the pseudo-labels as initial Ground Truth (GT); c. judging; d. obtaining semantic segmentation results, inputting the sub-images into a semantic segmentation network, and outputting semantic segmentation results; e. correcting and other steps, using a traditional pavement segmentation algorithm to obtain pseudo-labels, and continuously using conditional random fields to adjust the pseudo-labels during the training process, so that the pseudo-labels are infinitely close to Ground Truth (GT). It can avoid manual labeling, reduce workload, improve efficiency, and achieve the effect of improving pavement defect segmentation.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a road surface defect semantic segmentation method based on self-supervised learning. Background Art

[0002] In the field of autonomous driving, road conditions have an important impact on autonomous driving strategies.

[0003] The patent document with application number 202111144614.3 discloses a two-layer structure for semantic segmentation of the road surface: the first layer extracts features from the road surface image, and the second layer upsamples the features, restores them to the original image size, and realizes pixel classification of the original road surface image, thereby completing semantic segmentation. However, the data set adopted by the above scheme is completely manually labeled, which is a huge workload and inefficient. At present, there are only a few thousand open source labeled data sets at most, and the image environment taken is also quite different from the actual driving environment. In addition, this method has poor real-time performance and cannot meet the requirements of rapid road detection. Summary of the invention

[0004] In order to solve the current situation that it is difficult to obtain labeled data sets, the present invention aims to provide a pavement damage segmentation method based on self-supervised learning. Pseudo-labeling is obtained through a traditional pavement segmentation algorithm. During the training process, the pseudo-labeling is continuously adjusted using conditional random fields, so that the pseudo-label is infinitely close to the Ground Truth (GT) (real label, that is, the ideal segmentation result), avoiding manual labeling, reducing workload, improving efficiency, and achieving the effect of improving pavement damage segmentation.

[0005] To achieve the above object, the present invention is implemented by adopting the following technical solutions:

[0006] The method for semantic segmentation of pavement defects based on self-supervised learning disclosed in the present invention comprises the following steps:

[0007] a. Image cropping: randomly cropping the road image into several sub-images;

[0008] b. Generate pseudo labels. Use several semantic segmentation models to perform semantic segmentation on the sub-graphs, generate pseudo labels, and use the pseudo labels as the initial Ground Truth (GT).

[0009] c. Judgement: Train the sub-graph using a neural network to judge whether the sub-graph has a disease.

[0010] If the result of the judgment is that the disease exists, execute step d;

[0011] If the result of the judgment is that there is no disease, execute step g;

[0012] d. Obtain the semantic segmentation result, input the sub-graph into the semantic segmentation network, and output the semantic segmentation result;

[0013] e. Correction: Correct the semantic segmentation result through the conditional random field and combine it with the information of the prediction map generated last time to replace the initial Ground Truth (GT);

[0014] f. Repeat steps c, d, and e until the current prediction graph is the same as the previous prediction graph;

[0015] g. Take the average of the semantic segmentation results corresponding to all initial Ground Truth (GT) to obtain the final semantic segmentation result.

[0016] Furthermore, after step g, the method further includes:

[0017] h. Import the trained parameters into the Jetson nano development board.

[0018] Preferably, the neural network is a residual neural network.

[0019] Preferably, the several semantic segmentation models include at least two of the following semantic segmentation models:.

[0020] Furthermore, in step a, the road surface image is an original road surface image obtained with or without data enhancement processing.

[0021] Preferably, the data enhancement processing includes random contrast transformation and / or random brightness transformation.

[0022] Preferably, the image cropping uses a sliding window method to capture sub-images.

[0023] Preferably, in step h, the trained parameters are imported into a Jetson nano development board, and the Jetson nano development board has a semantic segmentation network solidified therein.

[0024] The beneficial effects of the present invention are as follows:

[0025] 1. The present invention gets rid of the manual labeling method and saves a lot of tedious labor by forming pseudo labels and gradually approximating the pseudo labels to real labels during training.

[0026] 2. The present invention adds a module for distinguishing whether there is a disease in the semantic segmentation network, which greatly improves the segmentation speed and achieves real-time performance.

[0027] 3. The present invention combines multiple semantic segmentation models to improve the robustness of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1It is a flow chart of the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings.

[0030] Example 1

[0031] like Figure 1 As shown, this embodiment discloses a road surface disease semantic segmentation method based on self-supervised learning, which is as follows:

[0032] At present, the data sets collected by cameras often have large image resolutions and require large memory space. Therefore, step a first randomly crops the road surface image, uses random contrast transformation, random brightness transformation and other methods to enhance the data, and cuts it into sub-images in the form of sliding windows. Step b first passes these sub-images through the semantic segmentation model to generate pseudo-labels, and regards these pseudo-labels as Ground Truth (GT). Step c puts these sub-images into the residual neural network for training to determine whether there are diseases in the sub-images. If there are, proceed to the next step; otherwise, it is directly attributed to the background, and steps d, e, and f are not performed. Step d puts the corresponding sub-image into the semantic segmentation network to obtain the semantic segmentation result. Step e corrects the generated semantic segmentation result through the conditional random field and combines it with the information of the prediction map generated last time, and then replaces it with the new GT (i.e., Ground Truth (GT)).

[0033] Step g repeats step d, step e, and step f until the generated semantic segmentation map does not change. Step g adds and averages the semantic segmentation results in step f corresponding to the initial GT generated by different semantic segmentation models in step b, thereby improving robustness.

[0034] Step h imports the trained parameters of the network into the development board to achieve realistic road surface disease segmentation.

[0035] Among them, the semantic segmentation model includes the following two types: a segmentation method based on the active contour model and an image segmentation model based on wavelet transform, as follows:

[0036] 1. Segmentation method based on active contour model: This method is a type of method that uses curve evolution to detect targets in a given image, based on which accurate edge information can be obtained. The basic idea is to first define the initial curve C, then obtain the energy function based on the image data, and then minimize the energy function to induce the curve to change, so that it gradually approaches the target edge and finally finds the target edge. The edge curve obtained by this dynamic approximation method has the advantages of being closed and smooth.

[0037] 2. Image segmentation model based on wavelet transform. Wavelet transform has the ability to detect local mutations of functions, so it can be used as an image edge detection tool. The edge of the image appears at the local gray discontinuity of the image, corresponding to the modulus maximum point of the wavelet transform. The edge wavelet transform of the image can be determined by detecting the modulus maximum points of the wavelet transform at each scale, and the wavelet transform at each scale can provide certain edge information. Therefore, multi-scale edge detection can be performed to obtain the ideal image edge, thereby segmenting the image.

[0038] This embodiment generates pseudo labels and continuously adjusts the pseudo labels during the training process, that is, the idea of ​​self-supervised learning, which can greatly reduce the workload of data set annotation. In actual work, road surface images can be randomly obtained from the area and put into the framework proposed by the present invention for training, so as to better meet the characteristics of road surface diseases in the area.

[0039] The area with road defects often accounts for a small proportion of all road surface areas, and the residual neural network model parameters are much smaller than the semantic segmentation network. Therefore, although adding a residual network to determine whether there are road defects will increase the model parameters overall, it will often jump out of the semantic segmentation network, reducing the running time on average and achieving better real-time effects than previous road segmentation models.

[0040] Example 2

[0041] This embodiment discloses a specific application example based on the present invention, which is as follows:

[0042] 1. Equip the car with a monocular camera, a Jetson nano development board, and a power supply.

[0043] 2. On a computer equipped with RTX3060Ti and running Ubuntu20.04, a large number of unlabeled road images are randomly cropped, and data enhancement is performed by random scaling, contrast changes, brightness changes, etc., and then divided into blocks through sliding windows, and then imported into the network designed in this patent.

[0044] 3. After multiple forward propagation and back propagation, when the network output results are stable, the network parameters obtained in the computer are imported into the Jetson nano development board.

[0045] 4. Acquire images through the monocular camera on the car and import the image data into the development board to achieve intelligent road semantic segmentation.

[0046] As can be seen from the above embodiments, the present invention can greatly reduce the amount of manual annotation, the required cost is very low, and the speed of semantic segmentation is also greatly improved. Since the final semantic segmentation result is an integration of multiple traditional semantic segmentation models, the robustness of the algorithm is also greatly improved. The present invention can provide a good solution to the problem of difficulty in timely detection and maintenance of road diseases caused by rapid traffic development.

[0047] Of course, the present invention may have many other embodiments. Without departing from the spirit and essence of the present invention, technicians familiar with the field may make various corresponding changes and deformations based on the present invention, but these corresponding changes and deformations should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. A road surface disease semantic segmentation method based on self-supervised learning. It is characterized in that The following steps are involved: a. Image cropping: randomly cropping the road image into several sub-images; b. Generate pseudo labels. Use several semantic segmentation models to perform semantic segmentation on the sub-graphs, generate pseudo labels, and use the pseudo labels as the initial Ground Truth (GT). c. Judgement: Train the sub-graph using a neural network to judge whether the sub-graph has a disease. If the result of the judgment is that the disease exists, execute step d; If the result of the judgment is that there is no disease, execute step g; d. Obtain the semantic segmentation result, input the sub-graph into the semantic segmentation network, and output the semantic segmentation result; e. Correction: Correct the semantic segmentation result through the conditional random field and combine it with the information of the prediction map generated last time to replace the initial Ground Truth (GT); f. Repeat steps c, d, and e until the current prediction graph is the same as the previous prediction graph; g. Take the average of the semantic segmentation results corresponding to all initial Ground Truth (GT) to obtain the final semantic segmentation result.

2. The method for semantic segmentation of pavement damage based on self-supervised learning according to claim 1, It is characterized in that After step g, the method further comprises: h. Import the trained parameters into the Jetson nano development board.

3. The method for semantic segmentation of pavement damage based on self-supervised learning according to claim 1 or 2, It is characterized in that The neural network is a residual neural network.

4. The method for semantic segmentation of pavement damage based on self-supervised learning according to claim 3, It is characterized in that The several semantic segmentation models include the following two: a segmentation method based on an active contour model and an image segmentation model based on wavelet transform.

5. The method for semantic segmentation of pavement damage based on self-supervised learning according to claim 3, It is characterized in that In step a, the road surface image is an original road surface image obtained with or without data enhancement processing.

6. The method for semantic segmentation of pavement damage based on self-supervised learning according to claim 5, It is characterized in that The data enhancement process includes random contrast transformation and / or random brightness transformation.

7. The method for semantic segmentation of pavement damage based on self-supervised learning according to claim 5 or 6, It is characterized in that The image cropping adopts a sliding window method to intercept sub-images.

8. The method for semantic segmentation of pavement damage based on self-supervised learning according to claim 1, It is characterized in that In step h, the trained parameters are imported into a Jetson nano development board, on which a semantic segmentation network is solidified.

Images