Road surface strip repair semantic segmentation method based on lightweight U-net

Abstract

The invention relates to a pavement strip repair semantic segmentation method based on lightweight U-net, and belongs to the technical field of semantic segmentation of deep learning. Aiming at the characteristic that in a pavement strip repair semantic segmentation data set picture, the composition of an asphalt pavement and strip repair diseases is relatively intuitive, a model pruning method is used to reduce a large number of parameters of a reference U-net model, and a strip repair semantic segmentation data set is used to train and obtain a lightweight U-net. From the experimental results of various evaluation indexes, the lightweight U-net model not only achieves the segmentation level of a mainstream semantic segmentation model on a pavement strip repair semantic segmentation data set, but also has remarkable lightweight advantages in the aspects of reasoning speed, parameter quantity, calculated quantity and reasoning speed indexes.

Description

Technical Field

[0001] The present invention relates to a road surface patch semantic segmentation method based on a lightweight U-net, belonging to the technical field of semantic segmentation of deep learning. Background Art

[0002] Asphalt pavement is the most widely used high-quality pavement in road construction, characterized by its impermeability, smooth dust removal, and durability. Strip patches are a type of pavement defect that can negatively impact the long-term performance of asphalt pavement. Semantic segmentation of these patches can accurately estimate their shape and area, enabling accurate maintenance. Current semantic segmentation models often suffer from large parameter counts, high computational complexity, and slow inference speed, making it difficult to achieve both fast and reliable semantic segmentation.

[0003] Current semantic segmentation detection methods are mainly divided into traditional methods and deep learning methods. Traditional methods include grayscale segmentation methods, conditional random field methods, etc.; deep learning methods include semantic segmentation methods based on deep learning.

[0004] The main ideas of each method are as follows:

[0005] (1) Grayscale segmentation method: This method is based on the threshold value of the pixel value intensity (grayscale intensity). One method using this technique is the Split and Merge algorithm. This algorithm recursively divides the image into sub-regions until a label can be assigned, and then combines adjacent sub-regions with the same label by merging them. It belongs to the traditional image semantic segmentation method.

[0006] (2) Conditional Random Field Method: CRF is a statistical modeling method for structured prediction. Unlike discrete classifiers, CRF can consider "adjacent context," such as the relationship between pixels, before making predictions, making it an ideal candidate for semantic segmentation. It also belongs to the traditional image semantic segmentation method.

[0007] (3) Semantic segmentation based on deep learning: Since the advent of FCN, semantic segmentation methods based on deep learning have been continuously developing and improving, becoming increasingly mature. This method extracts semantic information and recovers detailed features by training convolutional neural networks and using encoder and decoder structures, ultimately obtaining semantic segmentation results. It is one of the latest image semantic segmentation methods.

[0008] Problems with existing detection methods:

[0009] (1) Traditional image semantic segmentation methods, such as grayscale segmentation methods and conditional random field methods, can achieve good segmentation results in specific scenarios, but they may lack the ability to generalize to other scenarios. In unfamiliar or complex scenarios, it may be difficult to achieve ideal results and the results may be unstable.

[0010] (2) The latest image semantic segmentation methods, such as those based on deep learning, have good performance in most scenarios and are generally generalizable and stable. However, the disadvantage is that deep neural networks require a large amount of computation and parameters. Semantic segmentation models usually consist of an encoder and a decoder, so they usually require a device with strong computing power (such as a high-performance GPU) to run smoothly. Summary of the Invention

[0011] The purpose of the present invention is to provide a road surface patch semantic segmentation method based on lightweight U-net, aiming to solve the technical problems that the existing highway road surface patch semantic segmentation models generally have large computational complexity, long inference time and a large number of parameters.

[0012] To achieve the above objectives, the present invention provides a method for semantic segmentation of road patch patterns based on a lightweight U-net. This method first creates a semantic segmentation dataset for road patch patterns and then trains a lightweight U-net by improving the baseline U-net through model pruning. The lightweight U-net significantly reduces the number of parameters and computational complexity, while improving inference speed and achieving segmentation results that are consistent with or even better than mainstream semantic segmentation models. The specific steps include:

[0013] Step 1: Construct a semantic segmentation dataset of road surface patching;

[0014] Step 2: Replace the feature extraction backbone of the baseline U-net model and use model pruning to lightweight improve the baseline U-net model;

[0015] Step 3: Use the semantic segmentation dataset of road patching to train the improved lightweight U-net, and calculate various evaluation indicators of the lightweight U-net on the test set to obtain the optimal model parameters and perform semantic segmentation.

[0016] The Step 1 is specifically as follows:

[0017] Step 1.1: Obtain the grayscale image of the road surface crack size;

[0018] Step 1.2: Use semantic segmentation and annotation software to annotate the images and construct a semantic segmentation dataset of road patching.

[0019] Step 1.3: Divide the constructed semantic segmentation dataset into training set, validation set and test set in proportion.

[0020] The Step 2 is specifically as follows:

[0021] Step 2.1: Replace the feature extraction backbone of the baseline U-net model with the resnet50 feature extraction backbone;

[0022] Step 2.2: Use model pruning to perform lightweight improvements on the baseline U-net after replacing the feature extraction backbone.

[0023] The specific steps of Step 2.2 are as follows:

[0024] Improvement 1: In the encoder, the number of Bottlenets in the ResNet50 feature extraction backbone was reduced from 3+4+6+3=16 to 3+4=7.

[0025] Specifically, after the reduction, the encoder significantly reduces 18 1x1 convolution kernels, 9 3x3 convolution kernels, and 27 relu and batch normalization layers. Overall, the number of residual blocks is reduced by about 57%.

[0026] Improvement 2: In the skip connection, the feat1-feat5 feature map at the skip connection is reduced to feat1-feat3;

[0027] Specifically, after the reduction in the second improvement, the number of jump connection feature maps is reduced by 40%;

[0028] Improvement 3: Reduce the upsampling and feature fusion modules from 4 to 2.

[0029] Specifically, in the third improvement, since the decoder receives only feat1-3 skip connection feature maps, a total of two concact modules, four 3x3 convolution kernels, four relu layers and two upsampling layers for processing feat4 and feat5 are discarded, which reduces the size of the encoder by almost half compared to the original.

[0030] Furthermore, after completing the above three improvements, compared with the baseline U-net model, the network depth will be greatly reduced by about half, and the lightweight effect will be significant.

[0031] The Step 3 is specifically as follows:

[0032] Step 3.1: Set the training parameters, including batch size, feature extraction backbone name, number of categories, number of iterations, and number of threads;

[0033] Step 3.2: Use the semantic segmentation dataset of road patching to train and obtain a lightweight U-net model, and calculate various evaluation indicators of the lightweight U-net model to obtain the optimal model parameters for semantic segmentation.

[0034] The present invention offers the following benefits: It offers a lightweight improvement to the baseline U-Net, enabling it to achieve the same level of performance as mainstream semantic segmentation models in the semantic segmentation task of strip patching, while significantly reducing the number of parameters and computational overhead, and significantly improving inference speed. Compared to traditional semantic segmentation methods, deep learning-based semantic segmentation offers superior segmentation performance, greater reliability, and multi-scenario adaptability, significantly reducing labor, capital, and time costs while achieving stable and accurate segmentation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a flow chart of the steps of the present invention;

[0036] Figure 2 This is a diagram of the lightweight U-net network structure of the present invention;

[0037] Figure 3 It is the network structure diagram of U-net (resnet50) and U-net (vgg16) of the present invention;

[0038] Figure 4 This is a comparison chart of the parameters and computational complexity of the lightweight U-net of the present invention and the benchmark U-net;

[0039] Figure 5 This is a ranking chart of the inference speed of the lightweight U-net and other semantic segmentation models of the present invention;

[0040] Figure 6 This is a parameter / computation ranking diagram of the lightweight U-net and other semantic segmentation models of the present invention;

[0041] Figure 7 This is a comparative histogram of three evaluation indicators of the lightweight U-net of the present invention and other semantic segmentation models. Figure 8 This is a comparative histogram of three evaluation indicators of the lightweight U-net of the present invention and other semantic segmentation models on a small sample dataset. DETAILED DESCRIPTION

[0042] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0043] Example 1: Figure 1 As shown in the figure, a road patch semantic segmentation method based on lightweight U-net is proposed. The specific steps are as follows:

[0044] Step 1: Construct a semantic segmentation dataset of road surface patching.

[0045] Step 1.1: Obtain a grayscale image of the road surface patch with a size of 2048*2048. All images are captured and stored by a road image acquisition vehicle.

[0046] Step 1.2: Label the image using Labelme to construct a semantic segmentation dataset for road strip patches. Use polygons to outline the contours of the strip patches, where the strip patch damage is the foreground, the normal asphalt road surface is the background, and the labeled object is the strip patch.

[0047] Step 1.3: The image format of the dataset is all jpg, and the label format is all png. The images and labels of the standard strip patch semantic segmentation dataset are uniformly rotated 90°. Using data augmentation technology, the dataset is doubled to obtain 1000 images and corresponding labels. At the same time, a small sample strip patch semantic segmentation dataset is created, containing 250 images and corresponding labels.

[0048] Step 1.4: Divide the road surface crack sample set into a training set, a validation set, and a test set at a ratio of 8:1:1. In the present invention, 801 training sets, 99 validation sets, and 100 test sets are obtained.

[0049] Step 2: Replace the feature extraction backbone of the baseline U-net model and use model pruning to perform lightweight improvements on the baseline U-net model.

[0050] Step 2.1: Replace the feature extraction backbone of the baseline U-net model with the resnet50 feature extraction backbone;

[0051] Specifically, the baseline U-Net model in this paper uses the default feature extraction backbone of VGG16. Replacing VGG16 with the ResNet50 feature extraction backbone improves the model's computational load and inference speed, significantly reducing computational load by approximately 74% and increasing inference speed by approximately 90%. Subsequent improvements will be based on the U-Net network with the ResNet50 feature extraction backbone.

[0052] Step 2.2: Use model pruning to perform lightweight improvements on the baseline U-net after replacing the feature extraction backbone.

[0053] Improvement 1: In the encoder, the number of Bottlenets in the ResNet50 feature extraction backbone was reduced from 3+4+6+3=16 to 3+4=7.

[0054] Specifically, compared with the original 16 Bottlenet modules, the lightweight U-net has only 7 Bottlenet modules, which is only about 43% of the original number, more than half the number. After the reduction, the encoder has a total of 18 1x1 convolution kernels, 9 3x3 convolution kernels, and 27 ReLU and BN layers. The overall number of residual blocks is reduced by about 57%.

[0055] Improvement 2: In the skip connection, the feat1-feat5 feature map at the skip connection is reduced to feat1-feat3;

[0056] Specifically, after the reduction in the second improvement, the number of jump connection feature maps is reduced by 40%, providing data for subsequent feature fusion and upsampling operations;

[0057] Improvement 3: Reduce the upsampling and feature fusion modules from 4 to 2.

[0058] Specifically, in the third improvement, since the decoder receives only feat1-3 skip connection feature maps, a total of two concact modules, four 3x3 convolution kernels, four relu layers and two upsampling layers for processing feat4 and feat5 are discarded, which reduces the size of the encoder by almost half compared to the original.

[0059] Furthermore, after completing the above three improvements, compared with the baseline U-net model, the network depth will be greatly reduced by about half, and the lightweight effect will be significant.

[0060] Furthermore, in a dataset of 1000 standard sample strip inpainting, the following table compares the parameters of the lightweight U-Net and the baseline U-Net model using the VGG16 backbone and the ResNet50 backbone respectively:

[0061]

[0062] The network structures of the improved lightweight U-net model and the baseline U-net model are as follows: Figure 2 、 Figure 3 shown.

[0063] Step 3: Use the semantic segmentation dataset of road patching to train the improved lightweight U-net, and calculate various evaluation indicators of the lightweight U-net on the test set to obtain the optimal model parameters and perform semantic segmentation.

[0064] Step 3.1: Set the training parameters, including batch size, feature extraction backbone name, number of categories, number of iterations, and number of threads. The specific settings are as follows:

[0065]

[0066]

[0067] The environment configuration used for the training of the present invention is as follows:

[0068] cpu Intel i7-12800HX GPU Nvidia RTX 4070 Laptop framework PyTorch

[0069] Step 3.2: Train a lightweight U-net model on a 1000-image standard strip patching dataset and a 250-image small sample strip patching dataset, and calculate various U-net evaluation metrics, such as parameter count, computational effort, fps, mIoU, Recall / mPA, and mPrecision. The metrics for the lightweight U-net model in this invention are as follows:

[0070] Total GFLOPS 111.908G Totalparams 2.699M Fps 58fps mIoU (standard sample dataset / small sample dataset) 90.31％ / 89.67％ Recall / mPA (standard sample data set / small sample data set) 94.55％ / 94.21％ mPrecision (standard sample data set / small sample data set) 94.76％ / 94.37％

[0071] Step 3.3: Compare the indicators and parameters of the lightweight U-net and the U-net using two different backbones. The comparison results are as follows Figure 4 Compared to the baseline U-Net (vgg16), the parameter count has been significantly reduced by approximately 22 MiB, a decrease of approximately 89%. The computational effort has been reduced by approximately 340 GFOLPS, a decrease of approximately 75%. The inference speed has been increased by approximately 35 fps, a speedup of approximately 154%. Clearly, the lightweight U-Net has achieved significant improvements across all metrics.

[0072] Step 3.4: A comprehensive comparison is made between the obtained lightweight U-net indicators and the traditional semantic segmentation model, which shows that the lightweight U-net has advantages such as fast reasoning speed, small number of parameters, and high segmentation accuracy. The comparison of specific parameters and experimental results is as follows: Figure 5-Figure 8 shown.

[0073] Specifically, the number of parameters of the lightweight U-net is only 2.699M, which is about 6% and about 11% of the parameters of the baseline U-net model using the VGG and ResNet50 backbones, respectively, a reduction of about 94% and 89%; the computational complexity is only about 25% and about 60% of the latter two, a reduction of about 75% and 40%; in terms of model inference speed, the baseline U-net using the VGG and ResNet50 backbones are about 61% and about 26% slower than the lightweight U-net, respectively, while the latter is accelerated by about 154% and about 33%;

[0074] Furthermore, on the model evaluation metric Miou, the lightweight U-net not only maintains the same segmentation performance as the baseline U-net, but also comprehensively surpasses mainstream semantic segmentation models such as HRnet, deeplabv3+, PSPnet, and FCN in metrics such as mIoU on a standard 1000-image strip patching dataset. On a standard 250-image strip patching dataset, it even surpasses the segmentation accuracy of SegFormer. In summary, the lightweight U-net is a comprehensive semantic segmentation model that balances inference speed, low parameter and computational complexity, and high segmentation accuracy.

[0075] The above describes the specific embodiments of the present invention in detail with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary technicians in this field without departing from the scope of the present invention.

Claims

1. A road patch semantic segmentation method based on lightweight U-net, characterized by: The steps include: Step 1: Construct a semantic segmentation dataset of road surface patching; Step 2: Replace the feature extraction backbone of the baseline U-net model and use model pruning to lightweight improve the baseline U-net model; Step 3: Use the semantic segmentation dataset of road patching to train the improved lightweight U-net, and calculate various evaluation indicators of the lightweight U-net on the test set to obtain the optimal model parameters and perform semantic segmentation.

2. The method for semantic segmentation of road surface patches based on lightweight U-net according to claim 1, characterized in that: The Step 1 is specifically as follows: Step 1.1: Obtain the grayscale image of the road surface crack size; Step 1.2: Use semantic segmentation and annotation software to annotate the images and construct a semantic segmentation dataset of road patching. Step 1.3: Divide the constructed semantic segmentation dataset into training set, validation set and test set in proportion.

3. The method for semantic segmentation of road surface patches based on lightweight U-net according to claim 1, characterized in that: The Step 2 is specifically as follows: Step 2.1: Replace the feature extraction backbone of the baseline U-net model with the resnet50 feature extraction backbone; Step 2.2: Use model pruning to perform lightweight improvements on the baseline U-net after replacing the feature extraction backbone.

4. The method for semantic segmentation of road surface patches based on lightweight U-net according to claim 1, characterized in that: The specific steps of Step 2.2 are as follows: Improvement 1: In the encoder, the number of Bottlenets in the ResNet50 feature extraction backbone was reduced from 3+4+6+3=16 to 3+4=7. Improvement 2: In the skip connection, the feat1-feat5 feature map at the skip connection is reduced to feat1-feat3; Improvement 3: Reduce the upsampling and feature fusion modules from 4 to 2.

5. The method for semantic segmentation of road surface patches based on lightweight U-net according to claim 1, characterized in that: The Step 3 is specifically as follows: Step 3.1: Set the training parameters, including batch size, feature extraction backbone name, number of categories, number of iterations, and number of threads; Step 3.2: Use the semantic segmentation dataset of road patching to train and obtain a lightweight U-net model, and calculate various evaluation indicators of the lightweight U-net model to obtain the optimal model parameters for semantic segmentation.

Images