Lightweight road crack semantic segmentation algorithm fusing dense connection

Abstract

The invention provides a light-weight road crack semantic segmentation algorithm fusing dense connection. The algorithm is an improved segmentation algorithm based on a DeeplabV3 + network model. The structure of a network body is similar to that of a DeeplabV3 + network, a lightweight network is adopted as a network trunk, densely connected cavity space pyramid pooling layers are designed, the output of each layer serves as the input of each later layer, the feature information of cracks is overlapped layer by layer, and finally the crack features under multiple scales are gathered. The improved DeeplabV3 + network model can be used for more accurately segmenting a crack fuzzy object image, can improve the feature multiplexing efficiency while ensuring the overall segmentation precision, solves the problem of slow training time caused by the complex structure of the original DeeplabV3 + network, and effectively captures information blur and low segmentation accuracy caused by the complex environment.

Description

Technical Field

[0001] The present invention proposes a lightweight semantic segmentation algorithm for road cracks that integrates dense connections, and adopts a segmentation algorithm improved based on the DeeplabV3+ network model; the network body is similar to the DeeplabV3+ network structure, the network backbone adopts a lightweight network, and a densely connected void space pyramid pooling layer is designed. The output of each layer is used as the input of the subsequent layers, and the feature information of the cracks is superimposed layer by layer, and finally the crack features at multiple scales are gathered; the improved DeeplabV3+ network model can be used to more accurately segment crack blurred object images, which can improve the feature reuse efficiency while ensuring the overall segmentation accuracy, solve the problem of slow training time caused by the complex structure of the original DeeplabV3+ network, and effectively capture the complex environment that leads to blurred information and low segmentation accuracy. Background Art

[0002] The volume of highway transportation has been increasing rapidly. Through the use of roads over the years, as well as the impact of bad weather and natural disasters, various road pavement diseases will occur. Various hidden dangers will have a great impact on the stability of the road pavement structure, resulting in reduced pavement bearing capacity and durability, which can easily cause vehicle performance degradation, and ultimately affect driving safety. If not repaired in time, the degree of road damage will increase, resulting in higher road maintenance costs in the later stage, causing additional economic losses. Road maintenance is a way to prevent road damage and an important way to increase the service life of roads. Therefore, it is necessary to improve road maintenance management, which can not only reduce the occurrence of traffic accidents, but also reduce unnecessary economic losses. As a prerequisite for highway maintenance, road pavement disease detection has a relatively small impact on highway transportation in the early stage of pavement disease formation. Therefore, it is necessary to use effective methods to segment and identify road pavement diseases.

[0003] With the advancement of deep learning technology and the development of large-scale data sets, the semantic segmentation task has made rapid progress; in particular, the emergence of the Deeplab series of networks has made great progress in the semantic segmentation of road cracks; DeeplabV3+ is a context information aggregation semantic segmentation network based on spatial feature pyramids, which uses dilated convolution to obtain context information; DeeplabV3+ often generates a large number of parameters during operation, consumes a lot of running time, and only considers the segmentation accuracy without considering the real-time performance of the network. The application of road crack semantic segmentation requires not only segmentation accuracy, but also is very sensitive to the real-time performance of the algorithm. It requires the semantic segmentation algorithm to have real-time processing speed and fast interaction and response capabilities. Therefore, the key to the road crack segmentation algorithm is to ensure accuracy while improving the running speed. Summary of the invention

[0004] The technical problem solved by the present invention is to address the problems of low classification accuracy caused by incomplete semantic information and loose connection with context information in the DeeplabV3+ network, as well as the problems of large network parameters and slow running speed. A lightweight road crack semantic segmentation algorithm integrating dense connections is proposed. The designed global information extraction layer solves the problems of incomplete semantic information and loose connection with context information in the DeeplabV3+ network, thereby making the crack target segmentation effect better. The lightweight backbone network MobileNetV2 solves the problem of slow training speed caused by complex network structure, while also ensuring the target segmentation accuracy.

[0005] In order to achieve the above object, the technical solution of the present invention is as follows: A lightweight road crack semantic segmentation algorithm integrating dense connections includes the following steps: Step 1: Data preprocessing: change the data set to a size that the network can train and perform data enhancement according to the requirements; Step 2: Build a lightweight backbone network MobileNetV2; Step 3: Introduce a dense mechanism into the structure of spatial pyramid pooling; Step 4: Use the model to train the training set, obtain the road crack segmentation results, and save the best network model; Step 5: Load the network model and test the test set to obtain the road crack image segmentation results.

[0006] The specific situation in the step 1 is as follows:

[0007] A semantic segmentation dataset of road cracks is collected and divided into training set, validation set and test set.

[0008] Before model training, the Crack500 dataset was preprocessed and cropped into 16 non-overlapping sub-images (512*512, 512*512), and the images without background were filtered out, and finally 3367 images were synthesized. The dataset was expanded to 5000 images by rotation, horizontal flipping and translation methods, and the image size was finally unified to 512*512. The training set and validation set were divided into 8:2, and finally divided into 4000 training set images containing pavement crack targets and 1000 validation set images.

[0009] The specific situation in the step 2 is as follows:

[0010] Build a lightweight network MobileNetV2 as the backbone network to extract high-level semantic features.

[0011] The core operation of the MobileNetV2 network is to introduce the depthwise separable convolution replacement standard. The depthwise separable convolution has a more ideal and efficient control over the network parameter quantity and speed.

[0012] Depthwise separable convolution consists of two parts: depthwise convolution and pointwise convolution. Depthwise convolution is a convolution operation performed completely in a two-dimensional plane, with a one-to-one correspondence between channels and convolution kernels. Pointwise convolution is a common convolution with a convolution kernel size of 1x1, which is located after depthwise convolution and is used to fuse information from multiple channels and enhance the network's expressiveness.

[0013] During the convolution operation, if the number of input channels is , the convolution kernel size is , the number of output channels is , the output feature size is , then the ratio of the number of parameters of depth-separable convolution and standard convolution is as follows: ; The ratio of the calculated quantities is shown in the following formula: ; It can be seen from the two formulas that the computational complexity of depthwise separable convolution is much lower than that of standard convolution, meeting the requirements of small number of parameters and fast computing speed.

[0014] The MobileNetV2 network is composed of multiple inverted residual modules stacked together. The inverted residual modules help improve accuracy and build deeper networks. First, use Convolution increases the number of channels of the feature map, realizes the expansion of the feature map, enriches the number of features, and improves accuracy; secondly, use The depthwise separable convolution extracts the features of each channel, reducing the amount of computation. Finally, 1X1 convolution is used to reduce the number of channels. The activation function used after the dilated convolution and depthwise separable convolution is the Relu6 function, while the activation function after the compressed convolution is the Linear function, which prevents the Relu6 function from further destroying the compressed features.

[0015] The specific situation in step three is as follows:

[0016] A dense mechanism is introduced into the structure of spatial pyramid pooling, and the 1×1 ordinary convolution layer, three dilated convolution layers with dilation rates of 6, 12, and 18, and the global average pooling layer are improved from parallel connections to dense connections, so that the input of each layer is the output of all previous layers. The characteristic information of the cracks is superimposed layer by layer, and finally the crack features at multiple scales are gathered. The receptive field size of a single dilated convolution is shown in the formula: ; In the formula, is the receptive field obtained after convolution, is the convolution kernel size, is the void ratio, when When is 3 and the convolution kernel is also 3, the receptive field R is 7.

[0017] The size of the receptive field after the superposition of two dilated convolutions is shown in the formula: ; In the formula, is the new receptive field after superposition, and is the receptive field of the two dilated convolutional layers, is the number of dilated convolution layers. Under dilated convolution with the same dilated rate, the spatial pyramid pooling module with dense mechanism has a larger receptive field. The larger the receptive field, the more and more comprehensive the perceived content, which means that under the current dilated convolution, more feature information of the original image will be obtained by the model.

[0018] The specific situation in the step 4 is as follows:

[0019] Build a lightweight network model with multi-branch fusion.

[0020] Improvements are made on the original semantic segmentation DeeplabV3+ model, using the encoding and decoding framework and MobileNetV2 as the backbone network; in the encoder stage, the image is first extracted by the backbone network to extract complete information, and the generated high-level feature maps are sent to the dilated spatial pyramid pooling layer; the dilated spatial pyramid pooling layer consists of 3 dilated products with dilation rates of 6, 12, and 18, 1 1X1 convolution, and 1 global average pooling layer; then the 5 feature maps obtained are directly cascaded on the channel to complete the multi-scale sampling process; it can use dilated convolutions of different scales and an additional global average pooling to effectively extract key information and expand the receptive field; the fused feature map is connected to a 1x1 convolution to reduce its number of channels, and finally combined with the low-level feature Figure 1 The low-level feature maps provide detailed information, and the high-level feature maps provide semantic information.

[0021] In the decoder stage, a simple and efficient algorithm module is adopted. First, the high-level feature layer output by the decoder is bilinearly upsampled and enlarged to 4 times the original image. Next, the corresponding low-level feature layer with the same features in the feature extraction network backbone is convolved 1X1 to reduce the number of channels. Then the two feature layers are connected in series, and 3X3 convolution is used to refine the features. Finally, 4x upsampling is performed to complete the decoding operation.

[0022] The specific situation in the step five is as follows.

[0023] The training process uses the stochastic gradient descent optimization algorithm, setting the momentum to 0.9, the exponential decay rate of the second-order moment estimate to 0.999, the initial learning rate to 0.01, the learning rate weight decay to , the learning rate reduction method to select poly, and the learning rate reduction exponent to 0.9; the loss function uses the cross loss function based on the softmax function. The cross function is a loss function commonly used in processing classification problems. Its specific formula is: ; The softmax function processes the output results so that the sum of the predicted values ​​of multiple categories is 1, and then calculates the loss through crossover.

[0024] The data set is put into the network for training and evaluation to obtain the optimal network segmentation result, and the network model of the optimal network segmentation result is saved.

[0025] Test the test set and retain the test results and the generated crack segmentation map. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 The schematic diagram of the improved deplabv3+ model used in the present invention;

[0027] Figure 2 This is a structural diagram of the ASPP module with fused dense connections used in the present invention; DETAILED DESCRIPTION

[0028] For those skilled in the art, some well-known structures and their descriptions in the drawings may be omitted. The technical solution of the present invention is further described below in conjunction with the drawings and implementation methods.

[0029] The present invention aims to solve the problem that the crack feature extraction between scales generated by the current road crack semantic segmentation is incomplete, and the lost crack features cannot be effectively restored in the decoder part, and proposes a lightweight road crack semantic segmentation algorithm that integrates dense connections. First, the backbone network Xception is replaced with a lightweight Mobilenetv2 network to reduce model parameters and improve the expression ability of the model. Then, the ASPP module is selected to adopt a dense connection method to enhance the integrity of crack semantic segmentation. Dense connection is to use the output of each layer as the input of the subsequent layers, and the feature information of the cracks is superimposed layer by layer, and finally the crack features at multiple scales are gathered.

[0030] The specific implementation steps are: Step 1.1 Select a suitable road crack target dataset, such as the Crack500 dataset, and preprocess the dataset. Expand the dataset by cropping and affine transformation, and then crop it to 512*512 size. The expanded dataset has a total of 5,000 images, which are divided into training set and validation set according to 8:2; Step 1.2 Input the dataset images into the lightweight backbone network MobileNetV2, extract the shallow feature maps output by the first four layers of MobileNetV2 and the high-level feature maps output by the last layers: Step 1.3 Input the extracted high-level feature map into the dilated spatial pyramid pooling layer, and perform a fusion operation on the feature maps output by the two layers; Step 1.4 The feature map obtained after fusion is convolved to adjust the number of channels. The obtained feature map is first upsampled by 4 times, concatenated and fused with the shallow features obtained previously, and the final prediction map is obtained after 4 times upsampling to adjust the number of channels; Step 2.1 The multi-layer output high-level features obtained by the backbone network are compressed, activated, multiplied, etc., and their channel weight values ​​are assigned. Then, the features are input into each layer of dilated convolution of the spatial pyramid pooling module of the dense mechanism. The input of each layer of dilated convolution is the output of all previous layers. The number of input channels of each dilated convolution of the spatial pyramid pooling module of the dense mechanism is calculated as shown in the formula: ; In the formula, is the original number of deep feature channels, is the number of input channels corresponding to the dilated convolutional layers with different dilation rates; Step 2.2: After splicing and fusing the five feature maps obtained in the channel dimension, the feature map generated by the dense mechanism hollow space pyramid pooling module is obtained; Step 3.1 Use the improved lightweight DeeplabV3+ network model to train the dataset; Step 3.2 Input the fixed-size road crack image into the improved lightweight DeeplabV3+ network; Step 3.3 The MobileNetV2 network retrieves the trained model weights, preprocesses the image, extracts useful information from the image and generates a feature image; it is then transmitted to the improved dense mechanism’s dilated space pyramid pooling module and decoder part respectively; Step 3.4 The feature map processed by the encoder structure is upsampled 4 times by bilinear interpolation and fused with the shallow features. The generated feature map is further extracted using 3x3 convolution to obtain the fused feature map; Step 3.5 Upsample the fused feature map by 4 times to restore it to the original image size, output the prediction map, and complete the image segmentation; Step 4.1 sets the network's hyperparameters and the learning rate setting uses the Poly training strategy: Step 4.2: Put the data set into the network for training to obtain the optimal network segmentation result, and save the weight of the optimal network segmentation result; the training uses the cross entropy loss function CrossEntropyLoss, which is a commonly used loss function in dealing with classification problems; Step 4.3 loads the trained network model weights, puts the data set into the network for training and verification, and obtains the network segmentation result; saves the best segmentation weight for training the test set; obtains the segmentation result data and road crack segmentation map.

Claims

1. A lightweight road crack semantic segmentation algorithm integrating dense connections, comprising the following steps: Step 1: Data preprocessing: Change the data set to a size that can be trained by the network according to the requirements and perform data enhancement. Input the enhanced images into the constructed lightweight backbone network MobileNetV2. Step 2: Introduce a dense mechanism into the structure of spatial pyramid pooling; Step 3: Use the model to train the training set to obtain the road crack segmentation results, save the best network model, then load the network model, and test the test set to obtain the foggy street view image segmentation results.

2. According to the lightweight road crack semantic segmentation algorithm fused with dense connections according to claim 1, the specific process in Step 1 is as follows: Step 1.1 Select a suitable road crack Crack500 dataset and preprocess the dataset. Expand the dataset by cropping and affine transformation, and then crop it to 512*512 size. The expanded dataset has 5000 images in total, and is divided into training set and validation set according to 8:2; Step 1.2 Input the dataset images into the lightweight backbone network MobileNetV2, extract the shallow feature maps output by the first four layers of MobileNetV2 and the high-level feature maps output by the last layers: Step 1.3 Input the extracted high-level feature map into the dilated spatial pyramid pooling layer, and perform a fusion operation on the feature maps output by the two layers; Step 1.4 The feature map obtained after fusion is convolved to adjust the number of channels. The obtained feature map is first upsampled by 4 times, concatenated and fused with the shallow features obtained previously, and the final prediction map is obtained after 4 times upsampling to adjust the number of channels; According to the lightweight road crack semantic segmentation algorithm integrating dense connections according to claim 1, the specific process in Step 2 is as follows: Step 2.1 The multi-layer output high-level features obtained by the backbone network are compressed, activated, multiplied, etc., and their channel weight values ​​are assigned. Then, the features are input into each layer of dilated convolution of the spatial pyramid pooling module of the dense mechanism. The input of each layer of dilated convolution is the output of all previous layers. The number of input channels of each dilated convolution of the spatial pyramid pooling module of the dense mechanism is calculated as shown in the formula: ； In the formula, is the original number of deep feature channels, is the number of input channels corresponding to the dilated convolutional layers with different dilation rates; Step 2.2 concatenates and fuses the five feature maps obtained in the channel dimension to obtain the feature map generated by the dense mechanism void space pyramid pooling module.

3. According to the lightweight road crack semantic segmentation algorithm fused with dense connections as described in claim 1, the specific process in Step 3 is as follows: Step 3.1 Use the improved lightweight DeeplabV3+ network model to train the dataset; Step 3.2 Input the fixed-size road crack image into the improved lightweight DeeplabV3+ network; Step 3.3 The MobileNetV2 network retrieves the trained model weights, preprocesses the image, extracts useful information from the image and generates a feature image; it is then transmitted to the improved dense mechanism’s dilated space pyramid pooling module and decoder part respectively; Step 3.4 The feature map processed by the encoder structure is upsampled 4 times by bilinear interpolation and fused with the shallow features. The generated feature map is further extracted using 3x3 convolution to obtain the fused feature map; Step 3.5 Upsample the fused feature map by 4 times to restore it to the original image size, output the prediction map, and complete the image segmentation; step3.6 Put the data set into the network for training to obtain the optimal network segmentation result, and save the weight of the optimal network segmentation result; load the trained network model weights, put the data set into the network for training and verification, and obtain the network segmentation result; save the best one-time segmentation weight for training the test set; obtain the segmentation result data and road crack segmentation map.

Images