Road crack detection and segmentation method based on improved U-Net model

Abstract

The invention discloses a road crack detection and segmentation method based on an improved U-Net model, and the method comprises the steps: obtaining road crack images in different scenes, and carrying out the preprocessing and data set division of the road crack images; building an improved U-Net network model, training the improved U-Net network model by adopting the divided training set, inputting an image in a test set into the trained improved U-Net network model, obtaining a segmentation result, and segmenting cracks in the image; the improved U-Net network model is based on the original U-Net network model. The road crack detection accuracy and efficiency are improved, the safety and comfort of the road are ensured, and effective support is provided for road maintenance work.

Description

Technical Field

[0001] The present invention relates to a road crack detection and segmentation method based on an improved U-Net model. Background Art

[0002] Early road damage detection was mainly done by manual inspection and based on traditional machine vision algorithms. Manual methods are time-consuming, laborious, and inefficient, and are no longer sufficient to meet the increasing demand for road maintenance. Traditional machine vision-based image processing algorithms, such as median filtering, threshold algorithms, edge detection, etc., are sensitive to interference factors such as noise, lighting changes, and complex backgrounds in images, which can easily lead to instability and inaccuracy in processing results.

[0003] At present, the convolutional neural network model is commonly used in the pavement crack detection method based on deep learning. Since the convolutional neural network has the ability to autonomously learn features and translation invariance, it can overcome the problems that traditional digital image processing methods and machine learning are difficult to adapt to crack features and difficult to fit, and has broad application prospects in the field of image detection. However, the current pavement image crack detection model, such as the existing patents: CN110569730A, CN115035371A, CN115953672A, does not effectively distinguish between crack areas and backgrounds, and there are problems in the crack data set such as large intra-class distances, small inter-class distances, and discontinuous and blurred boundaries of crack areas. Summary of the invention

[0004] The present invention aims to solve the problems existing in the above-mentioned prior art and provide a road crack detection and segmentation method based on an improved U-Net model.

[0005] The technical solutions adopted in the present invention are:

[0006] A road crack detection and segmentation method based on an improved U-Net model includes the following steps:

[0007] S1: Obtain road crack images in different scenarios, preprocess the road crack images, and divide the data sets;

[0008] S2: Build an improved U-Net network model, use the divided training set to train the improved U-Net network model, input the image in the test set into the trained improved U-Net network model, obtain the segmentation result, and segment the cracks in the image; the improved U-Net network model is based on the original U-Net network model, and the improvements are as follows:

[0009] (1) In the backbone network of the original U-Net network model, the pre-trained DeseNet-121 is used as the feature extractor;

[0010] (2) In the encoder of the original U-Net network model, a multi-layer feature extraction and downsampling process is constructed through dense connections and transition layers;

[0011] (3) In the decoder of the original U-Net network model, feature resolution is gradually restored through upsampling and skip connections;

[0012] (4) A two-layer spatial attention module is introduced at the transition position between the encoder and the decoder, and an improved dilated spatial pyramid pooling module is added after the last convolutional layer of the decoder. The ASPP module is used to replace the skip connection at the bottom layer of the decoder.

[0013] Furthermore, the two-layer spatial attention module is improved based on the CBAM attention mechanism, and the improvements are as follows:

[0014] The channel attention module of the CBAM attention mechanism is replaced with a two-level attention module, and a cross-channel interaction mechanism is introduced in the two-level attention module, and a hole convolution mechanism is introduced in the spatial attention module of the CBAM attention mechanism.

[0015] Furthermore, the improved atrous spatial pyramid pooling module is improved on the basis of the original ASPP module, and the improvements are: adding a new branch to the ASPP module and introducing a global pooling operation;

[0016] Furthermore, the Transition Layer includes an average pooling layer and a 1x1 convolutional layer.

[0017] Furthermore, the encoder includes a plurality of dense module DenseBlock structures, and a Dropout layer is added to each dense module DenseBlock structure.

[0018] Furthermore, a training set is used to train the improved U-Net network model, and a loss function is used to update corresponding parameters of the improved U-Net network model to train and segment the improved U-Net network model. The loss function combines a binary cross entropy loss function with a Dice coefficient loss function.

[0019] Furthermore, the improved U-Net network model was evaluated and verified using intersection-over-union ratio, precision, accuracy, recall rate and F1 value as performance evaluation indicators.

[0020] The present invention aims to solve the problems of existing deep learning models, such as discontinuous crack extraction, inability to solve the problem of sample imbalance, poor model generalization ability and robustness, huge amount of required training samples, and low model calculation efficiency that cannot perform real-time detection. A road crack detection and segmentation method based on an improved U-Net model is proposed. The present invention adds dense connections to the original connections, and through the stacking of multiple convolutional layers, the network can fully utilize the feature information of the previous layer, and pass the features to the subsequent layers through jump connections, thereby enhancing feature reuse and information fluidity. A dual-layer spatial attention module (DCSA) is introduced at the transition position between the encoder and the decoder, including a dual-level attention module (BiLevel Attention) and a spatial attention module (Spatial Attention). The two-level attention module can improve the representation ability and discrimination of features. The spatial attention module learns the spatial correlation of feature maps by average pooling and maximum pooling operations on feature maps. The improved atrous spatial pyramid pooling module (ASPPPlus module) is added after the last convolutional layer of the decoder. By using atrous convolutions of different scales, the receptive field is increased and contextual information of different scales is captured. The ASPP module is added at the bottom of the network to further improve the model's perception of crack features of different scales. The method has better detection and segmentation accuracy for road cracks than the traditional U-net. The improved network provides a feasible solution to the problem of road crack detection and segmentation.

[0021] The present invention improves the accuracy and efficiency of road crack detection to ensure road safety and comfort and provide effective support for road maintenance work. Dense connections alleviate the gradient vanishing problem to a certain extent, promote the flow of information and reuse of features, and help extract richer features. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 To improve the network structure diagram of U-Net network architecture;

[0023] Figure 2 Figure 2 is the structural diagram of the double-layer spatial attention module.

[0024] Figure 3 Structural diagram of the improved atrous spatial pyramid pooling module. DETAILED DESCRIPTION

[0025] The present invention will be further described below in conjunction with the accompanying drawings.

[0026] like Figures 1 to 3 The present invention provides a road crack detection and segmentation method based on an improved U-Net model, comprising the following steps:

[0027] S1: Obtain road crack images in different scenarios, preprocess the road crack images, and divide the data sets.

[0028] S2: Build and improve the U-Net network architecture, specifically:

[0029] (1) Based on the original U-Net network model, the characteristics of the encoder and decoder are combined. In the backbone network, the pre-trained DeseNet-121 is used as the feature extractor;

[0030] (2) In the encoder part, a multi-layer feature extraction and downsampling process is constructed through dense connections and transition layers to alleviate the problem of gradient disappearance;

[0031] (3) In the decoder part, the features are gradually restored to a higher resolution through upsampling and skip connections, which improves the prediction accuracy.

[0032] (4) A double-layer spatial attention module (DCSA) is introduced at the transition position between the encoder and the decoder; an improved atrous spatial pyramid pooling module (ASPPPlus module) is added after the last convolutional layer of the decoder, and an ASPP module is added at the bottom of the network.

[0033] S3: The improved U-Net network model is trained using the training set road crack images.

[0034] S4: Input the road crack image in the test set into the trained U-Net network model to obtain the segmentation result.

[0035] In S1, obtaining the road crack image and preprocessing it specifically includes:

[0036] 1) Dataset screening:

[0037] The open source crack image datasets (CRACK500, CrackForest, CrackMap, DeepCrack, etc.) were obtained and screened to construct a suitable crack image dataset totaling 1,403 images.

[0038] 2) Dataset trimming:

[0039] When automatically cropping the original data set, the selected cropping box is located at the center of the original image, and the cropped crack image will be scaled to a size of 256*256 pixels.

[0040] 3) Image calibration, data enhancement and data partitioning:

[0041] The image calibration manually marks the target pixels as 255 grayscale values ​​and the background as 0 grayscale value; data enhancement uses flip transformation, rotation transformation and brightness contrast transformation; data partitioning randomly divides the image after data enhancement operation into a training set accounting for 70%, a validation set accounting for 20%, and a test set accounting for 10%.

[0042] The specific steps to enhance data using flip transformation, rotation transformation and brightness contrast transformation in the above steps are:

[0043] (1) Flip transformation: The original image is flipped horizontally or vertically with a probability of 50%. The change matrix from the original pixel (x0, y0) to the transformed point (x, y) after horizontal and vertical mirroring is:

[0044]

[0045] (2) Rotation transformation: Rotate the original image clockwise around the center by an angle of θ. The change matrix from the original pixel (x0, y0) to the transformed point (x, y) after rotation transformation is:

[0046]

[0047] (3) performing one of the hue-saturation-value transformation, brightness transformation, and contrast transformation operations; the hue-saturation-value transformation operation randomly adjusts the hue, saturation, and value of the image to increase the diversity of the image; the brightness transformation operation randomly adjusts the brightness of the image to increase the diversity of the image; and the random contrast transformation operation randomly adjusts the contrast of the image to increase the diversity of the image.

[0048] The improved U-Net network architecture constructed by the present invention is further described below.

[0049] The pre-trained DeseNet-121 is used as the feature extractor. The encoder includes a dense module DenseBlock structure and a Transition Layer. The dense module DenseBlock structure consists of multiple densely connected layers, and each densely connected layer consists of batch normalization (BatchNorm), ReLU activation function, convolution layer and Dropout layer.

[0050] The Transition Layer contains a 1x1 convolution layer and an average pooling layer to reduce the size of the feature map.

[0051] The decoder includes deconvolution (upsampling) and dense module DenseBlock structure.

[0052] The double-layer spatial attention module (DCSA) is added at the transition position between the encoder and the decoder. The double-layer spatial attention module is improved based on the CBAM attention mechanism. The improvements are:

[0053] The channel attention module of the CBAM attention mechanism is replaced with a two-level attention module, and a cross-channel interaction mechanism is introduced in the two-level attention module, and a hole convolution mechanism is introduced in the spatial attention module of the CBAM attention mechanism.

[0054] DCSA includes a bi-level attention module (BiLevel Attention) and a spatial attention module (SpatialAttention) to enhance the network's receptive field and contextual information.

[0055] Among them, the two-level attention module learns the global information of the feature map through average pooling and maximum pooling operations, and obtains the attention map through convolution operation fusion. The cross-channel interaction mechanism is introduced in the two-level attention module to enhance the information flow between channels.

[0056] The spatial attention module learns the spatial distribution information of the feature map through average pooling and maximum pooling operations, and obtains the attention map by replacing the convolution with the dilated convolution operation. In order to further improve the stability and performance of the model, the residual connection and batch normalization technology are introduced in the DCSA module. Among them, the ratio parameter used by the two-level attention module to calculate the channel number reduction is set to 4; the convolution kernel parameter in the spatial attention module is set to 5.

[0057] The improved atrous spatial pyramid pooling module (ASPPPlus module) is improved on the basis of the original ASPP module. The improvements are: adding branches to the ASPP module and introducing global pooling operations.

[0058] The ASPPPlus module is added after the last convolutional layer of the decoder. It uses dilated convolution (with different sampling rates) and global pooling to fuse multi-scale receptive fields. The ASPP module is used to replace the jump connection at the bottom layer of the decoder to further improve the model's perception of crack features at different scales.

[0059] A Dropout layer is added to each layer in a DenseBlock. Specifically, a Dropout layer is added after each convolutional layer in each DenseBlock. The Dropout rate parameter in this invention is set to 0.2, which means that the output of each convolutional layer in each DenseBlock has a 20% probability of being set to zero during training.

[0060] Final output: Map the feature map to a feature map of the specified number of categories through a convolutional layer.

[0061] In the present invention, the training set road crack images are used to train the improved U-Net network model, specifically including:

[0062] (1) The hyperparameters of the training process are: initial learning rate = 0.001, number of training iterations = 100, and number of samples per training = 3.

[0063] (2) The optimizer selects adaptive moment estimation to update network parameters. Adaptive moment estimation Adam is a commonly used optimization algorithm used to train neural network models and other machine learning models.

[0064] Input the images in the test set into the trained improved U-Net network model to obtain the segmentation results, as follows:

[0065] The evaluation parameters used are intersection over union (IoU), precision, accuracy, recall, and F1 score. The calculation formula for IoU is as follows:

[0066]

[0067] Where A: crack area pixel set, B: annotated crack area pixel set;

[0068] The calculation formula of precision is as follows:

[0069]

[0070] The calculation formula of accuracy is as follows:

[0071]

[0072] The calculation formula of recall is as follows:

[0073]

[0074] The calculation formula of F1 Score is as follows:

[0075]

[0076] In the formula, TP is the correctly predicted positive example. That is, the true value of the data is a positive example, and the predicted value is also a positive example; TN is the correctly predicted negative example. That is, the true value of the data is a negative example, and the predicted value is also a negative example; FP is the incorrectly predicted positive example. That is, the true value of the data is a negative example, but it is incorrectly predicted as a positive example; FN is the incorrectly predicted negative example. That is, the true value of the data is a positive example, but it is incorrectly predicted as a negative example.

[0077] Based on the original U-Net network model, the features of the encoder and decoder are combined. In the backbone network, the pre-trained DeseNet-121 is used as the feature extractor; in the encoder part, a multi-layer feature extraction and downsampling process is constructed through dense connections and TransitionLayer to alleviate the problem of gradient disappearance; in the decoder part, the features are gradually restored to resolution through upsampling and skip connections, which improves the prediction accuracy; a double-layer spatial attention module (DCSA) is introduced at the transition position between the encoder and decoder; and an improved atrous spatial pyramid pooling module (ASPPPlus module) is added after the last convolutional layer of the decoder. In general, the above network model enhances the network's feature extraction ability, perception ability and semantic expression ability by introducing the DenseNet backbone network, DCSA module and ASPPPlus module, etc., combined with the deep supervision mechanism, and effectively improves the recognition accuracy of road cracks in image segmentation tasks.

[0078] After the above steps, the road crack image is detected and segmented, and the evaluation parameters are compared with the segmentation evaluation parameters of several other existing methods. The evaluation parameters of this method are better, the processing time is faster, and the robustness is better, which shows the feasibility and effectiveness of the method of the present invention.

[0079] The above description is only a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements without departing from the principle of the present invention, and these improvements should also be regarded as within the protection scope of the present invention.

Claims

1. A road crack detection and segmentation method based on an improved U-Net model, characterized in that: The following steps are involved: S1: Obtain road crack images in different scenarios, preprocess the road crack images, and divide the data sets; S2: Build an improved U-Net network model, use the divided training set to train the improved U-Net network model, input the image in the test set into the trained improved U-Net network model, obtain the segmentation result, and segment the cracks in the image; the improved U-Net network model is based on the original U-Net network model, and the improvements are as follows: (1) In the backbone network of the original U-Net network model, the pre-trained DeseNet-121 is used as the feature extractor; (2) In the encoder of the original U-Net network model, a multi-layer feature extraction and downsampling process is constructed through dense connections and transition layers; (3) In the decoder of the original U-Net network model, feature resolution is gradually restored through upsampling and skip connections; (4) A two-layer spatial attention module is introduced at the transition position between the encoder and the decoder, and an improved dilated spatial pyramid pooling module is added after the last convolutional layer of the decoder. The ASPP module is used to replace the skip connection at the bottom layer of the decoder.

2. The road crack detection and segmentation method based on the improved U-Net model according to claim 1, characterized in that: The two-layer spatial attention module is improved based on the CBAM attention mechanism. The improvements are as follows: The channel attention module of the CBAM attention mechanism is replaced with a two-level attention module, and a cross-channel interaction mechanism is introduced in the two-level attention module, and a hole convolution mechanism is introduced in the spatial attention module of the CBAM attention mechanism.

3. The road crack detection and segmentation method based on the improved U-Net model as claimed in claim 1, characterized in that: The improved atrous spatial pyramid pooling module is improved on the basis of the original ASPP module, and the improvements are: adding a new branch to the ASPP module and introducing a global pooling operation.

4. The road crack detection and segmentation method based on the improved U-Net model as claimed in claim 1, characterized in that: The Transition Layer includes an average pooling layer and a 1x1 convolution layer.

5. The road crack detection and segmentation method based on the improved U-Net model as claimed in claim 1, characterized in that: The encoder includes a plurality of dense module DenseBlock structures, and a Dropout layer is added to each dense module DenseBlock structure.

6. The road crack detection and segmentation method based on the improved U-Net model as claimed in claim 1, characterized in that: The improved U-Net network model is trained with a training set, and the corresponding parameters of the improved U-Net network model are updated with a loss function to train and segment the improved U-Net network model. The loss function combines a binary cross entropy loss function with a Dice coefficient loss function.

7. The road crack detection and segmentation method based on the improved U-Net model as claimed in claim 1, characterized in that: The improved U-Net network model is evaluated and verified using intersection-over-union ratio, precision, accuracy, recall rate and F1 value as performance evaluation indicators.

Images