Asphalt pavement crack detection method based on improved DeepLabv3+ network

Abstract

The application belongs to the field of deep learning, and particularly relates to a kind of semantic segmentation method based on improved DeepLabv3+ network, comprising: obtaining asphalt road crack data image by shooting device;The road crack data information is pretreated, and the pretreated data is input into the trained semantic segmentation network, and the crack state information is obtained;The severity of crack is determined according to the crack state information;The semantic segmentation network comprises: input pretreatment module, improved ASPP module, attention mechanism module, output convolution calculation module and output post-processing module;The semantic segmentation network of the application is optimized through ECA attention module, CARAFE up sampling and improved ASPP module, so as to improve the accuracy of model recognition and the algorithm generalization ability.

Description

TECHNICAL FIELD

[0001] The present application relates to a kind of road crack identification method based on improved DeepLabv3+ network, it is related to image processing technology, deep learning, neural network and other fields. BACKGROUND

[0002] Road inspection is an important part of daily maintenance work, and plays an important role in the stable development of China's economy. Due to the complex and changeable road crack situation, manual inspection is low in efficiency and easily affected by subjective factors, which cannot meet the growing demand. In recent years, with the frequent use of roads, it is inevitable to have more or less crack problems, which has brought great security risks to traffic development.

[0003] At present, people have proposed various deep learning semantic segmentation models, such as PSPNet, U-Net, SegNet and DeepLab series; PSPNet network uses different pooling operations to control the receptive field, which plays a certain role in improving the detection speed, but cannot fully utilize the scene, and its detection accuracy is affected; U-Net introduces a skip connection structure, which connects the corresponding bottom features to the deconvolution layer in the decoding process, so that the model can fully utilize the semantic features of each layer, and can obtain better segmentation results even in the case of less training samples, but the boundary detection ability of this model is weak, which may lead to inaccurate boundary detection results; SegNet uses VGG-16 as the encoder layer, and each encoder layer corresponds to a decoder layer, and the output of the final decoder is sent to the soft-max classifier to independently generate class probability for each pixel, but its segmentation result is relatively rough; DeepLabv3+ belongs to the DeepLab series, which combines the spatial pyramid (Atrous Spatial Pyramid Pooling, ASPP) module and the encoder-decoder architecture to capture multi-scale features and obtain more accurate object boundaries, so more accurate detection results can be obtained. SUMMARY

[0004] To solve the problems existing in the prior art, the present application proposes a road crack detection method based on improved DeepLabv3+ model, which comprises: acquiring asphalt road crack data images by using a shooting device; preprocessing the road crack data information, and inputting the preprocessed data into the trained semantic segmentation network to obtain crack state information; determining the severity of the crack according to the crack state information; the semantic segmentation network comprises: an input preprocessing module, an improved ASPP module, an attention mechanism module, an output convolution calculation module and an output post-processing module.

[0005] Training the semantic segmentation network comprises: acquiring crack data, marking the acquired data to obtain labeled road crack data; the labeled crack data is normalized; the normalized data is preprocessed by using an input preprocessing module; the preprocessed data is input into an improved ASPP module for feature extraction after generating an effective feature layer by a backbone network; the channel attention module is used to learn the correlation between channels and adaptively adjust the weight of the channel; after obtaining the final effective layer, the output convolution calculation module is used to adjust the number of channels to obtain a segmentation result; the output post-processing module is used to convert the segmentation result into a probability value; the model loss function is calculated according to the probability value, and the model parameters are adjusted, and the training of the model is completed when the loss function converges.

[0006] The beneficial effects of the present application are as follows:

[0007] The advantages of the present application are as follows: 1. By using the lightweight backbone network MobileNetv2, the model complexity is reduced, and the model is more lightweight; 2. By introducing the channel-based ECA attention module, the module learns the correlation between channels and adaptively adjusts the weight of the channel to improve the network performance; 3. The CARAFE up-sampling is introduced to capture the rich semantic information required for dense prediction tasks, effectively reducing the false positive rate; 4. The improved ASPP module is used to further enhance the feature extraction capability of the model; 5. The deep learning technology is used to identify the cracks of the asphalt road, avoiding the problems caused by the inconvenience of traditional manual crack detection; 6. The method and device are simple, and the acquisition device only needs two cameras installed at the tail of the vehicle to obtain image data, and after subsequent network training, the crack detection can be obtained. BRIEF DESCRIPTION OF DRAWINGS

[0008] Figure 1 Improved DeepLabv3+ network structure diagram

[0009] Figure 2 Partial data set expansion result diagram

[0010] (a) is the original picture of the crack

[0011] (b) is the picture after up-down flipping

[0012] Figure 3 DeepLabv3+ network structure diagram

[0013] Figure 4 MobileNetv2 network structure diagram

[0014] Figure 5 ECA network structure diagram

[0015] Figure 6 CARAFE up-sampling network structure diagram

[0016] Figure 7 For improved ASPP network structure diagram

[0017] Figure 8 For model training results

[0018] Figure 9 For road crack original picture

[0019] (a) is a transverse crack original picture

[0020] (b) is a longitudinal crack original picture

[0021] (c) is a mesh crack original picture

[0022] Figure 10 For model identification results

[0023] (a) is a transverse crack identification result picture

[0024] (b) is a longitudinal crack identification result picture

[0025] (c) is a mesh crack identification result picture DETAILED DESCRIPTION

[0026] The application provides a road crack detection method based on an improved DeepLabv3+ network model, in order to make the technical scheme and effects of the application clearer and more explicit, the specific embodiments of the application are described in detail below with reference to the drawings.

[0027] A road crack detection method based on an improved DeepLabv3+ model, the method comprises: acquiring an asphalt road crack data image by using a shooting device; preprocessing data information of the road crack, inputting the preprocessed data into a trained semantic segmentation network, and obtaining crack state information; determining the severity of the crack according to the crack state information;

[0028] The structure of the improved DeepLabv3+ is as designed Figure 1The application designs a road crack detection method based on an improved DeepLabv3+ network model, which includes three stages: a dataset making stage, a model recognition stage, and a model prediction stage. The dataset making stage mainly manually labels the data pictures of the actually photographed road crack pictures to obtain corresponding label pictures. The model recognition stage is mainly based on an improved DeepLabv3+ network structure, which is composed of an encoder and a decoder. After the input picture passes through the backbone network, the corresponding deep features and shallow features are output. The deep features are sent into the improved ASPP and then into the channel-based ECA attention module, and finally, 1×1 convolution operation is performed to obtain the multi-scale context information. In the decoding stage, the deep semantic features from the encoding stage are first subjected to CARAFE up-sampling, and then fused with the shallow features sent into the ECA module, and 3×3 convolution operation is performed. CARAFE up-sampling obtains the final segmentation result. The prediction stage mainly sends the pictures without labeling into the model recognition to finally obtain the segmentation result.

[0029] 1. Dataset making stage: Since the original dataset is very small, data set expansion is needed before being sent into the model training. The expansion method mainly includes up-down flipping, adding public datasets, and increasing negative samples. In order to better adapt to the training effect of the model, the dataset used in this patent is sent into the model training after being cropped. When the image is segmented, it is divided into two categories of cracks and backgrounds, and each picture has a corresponding manual annotation. The expanded dataset is as shown in Figure 2 .

[0030] 2. Original DeepLabv3+ model: The network structure of the original DeepLabv3+ model is as shown in Figure 3 . In the Encoder structure, the input image is first subjected to feature extraction by the backbone network (Xception), and the obtained feature images are transmitted to the Atrous Spatial Pyramid Pooling (ASPP) and Decoder structure. The ASPP structure uses different hole convolution layers with different hole rates to obtain the multi-scale information of the input feature map and is spliced and fused to obtain the high-level semantic features of the image, and then uses 1×1 convolution to adjust the output channel number. In the Decoder structure, the high-level semantic features output by the Encoder structure are subjected to 4 times up-sampling to restore the resolution to 1 / 4 of the original image, and the low-level features output by the Backbone network are subjected to 1×1 convolution to adjust the channel number, and are fused with the high-level semantic feature map. After that, the fused feature map is subjected to 3×3 convolution operation and 4 times up-sampling to obtain the prediction result.

[0031] 3. Main network MobileNetv2: MobileNetv2 has a more streamlined network structure, which is a lightweight network model known and used for its small number of parameters and high prediction accuracy. The basic structure of Mobilenetv2 is composed of three parts: depth separable convolution, reverse residual and linear bottleneck structure. The most highlight is the reverse residual structure, and in the reverse residual structure, the last layer of the convolution layer uses a linear activation function. Its network structure is as shown in Figure 4 .

[0032] 4. ECA attention mechanism module: ECA is an efficient channel attention module, which only designs a few parameters, but has obvious effect gain. Appropriate cross-channel interaction can significantly reduce model complexity while maintaining performance, and the non-reduced local cross-channel interaction strategy effectively avoids the influence of dimension reduction on channel attention learning effect. The overall structure is as shown in Figure 5 , which operates on feature maps, extracts more refined features through channel attention modules, and thus improves the expressiveness of the model.

[0033] 5. CARAFE upsampling: the most widely used feature upsampling operator is the nearest neighbor and bilinear interpolation, which uses the spatial distance between pixels to guide the upsampling process. However, nearest neighbor interpolation and bilinear interpolation only consider sub-pixel neighborhoods and cannot capture rich semantic information required for dense prediction tasks. The CARAFE upsampling used in this patent is a general, lightweight and efficient operator that can aggregate context information within a larger receptive field, dynamically adapt to instance-specific content and maintain computational efficiency. The network structure is as shown in Figure 6 . CARAFE is mainly composed of an upsampling kernel prediction module and a feature reorganization module. At each location, CARAFE can use the underlying content information to predict a reorganization kernel and reorganize features within a predefined neighborhood.

[0034] 6. Feature extraction stage: the feature extraction stage mainly uses dilated convolution with different dilated rates for feature extraction to improve the receptive field of the network, so that the network has different feature receptive situations. The improved ASPP structure diagram is as shown in Figure 7 . The atrous convolution layer is organized in a cascaded manner, where the dilated rate of each layer increases layer by layer. The layer with small dilated rate is placed at the lower part, and the layer with large dilated rate is placed at the upper part. The output of each atrous layer is concatenated with the input feature map and all outputs from lower layers, and the concatenated feature map is fed to the next layer; the final output of DenseASPP is the feature map generated by multi-rate and multi-scale atrous convolution. The proposed structure can form a denser and larger feature pyramid at the same time, using only a few atrous convolution layers; each atrous layer in DenseASPP can be represented as:

[0035]

[0036] where d l denotes the dilation rate of the l-th layer, [···] denotes concatenation operation, [y l-1 ,y l-2 ,…,y0] denotes the feature map formed by concatenating the outputs of all previous layers; compared with the original ASPP, DenseASPP stacks all dilated layers together and connects them through dense connections, which brings us two benefits: a denser feature pyramid and a larger receptive field. The present application adds SP strip pooling to the DenseASPP, which deploys a long strip-shaped pooling kernel shape along the spatial dimension, so it can capture the long-distance relationship of isolated regions, while keeping a narrow kernel shape in other spatial dimensions, which is convenient for capturing local context and preventing irrelevant regions from interfering with label prediction.

[0037] 7. Loss function: Traditional loss functions such as cross-entropy, mean square, etc. have poor performance in the case of data class imbalance. The smaller class will be overwhelmed by the larger class, resulting in high prediction accuracy during training but poor application efficiency. Since most of the road is intact, only a small part is cracked, and these small parts are mostly caused by frequent squeezing of large public transportation tools, which causes a problem of foreground-background imbalance, that is, most of the area in the picture does not contain the target, only a small part of the area contains the target; however, the Dice Loss function can effectively alleviate the negative effects of foreground-background imbalance, so we introduce the Dice Loss function into the improved DeepLabv3+ model. The loss of the training set of the network is shown in Figure 8

[0038] In binary classification, cross-entropy is often used to evaluate the difference between labels and predictions. The calculation is shown in equation (2), where y represents the label, y' represents the prediction, and N represents the number of pixels

[0039]

[0040] Dice Loss is named after the Dice coefficient, which is a measure of the similarity between two samples. The larger the value, the more similar the two samples are. The calculation of the Dice coefficient is shown in equation (3):

[0041]

[0042] ​where X represents the true case of the crack, Y represents the predicted case. |X∩Y| represents the number of foreground intersection pixels between X and Y, |X| and |Y| represent the number of pixels of foreground (crack) in the specific image in the label and prediction respectively. The Dice Loss expression is shown in equation (4):

[0043]

[0044] As shown in equation (5), the final loss function takes the form of the sum of the cross-entropy function and the Dice loss function.

[0045] L = L ce + L dice (5)

[0046] 8. Model prediction stage: finally, the unlabeled original crack picture is taken as the test set into the model prediction, and the obtained segmentation result is shown in Figure 9 . Figure 10 .

Claims

1. A road crack detection method based on an improved DeepLabv3+ model, characterized in that, The application relates to an asphalt road crack detection method based on a semantic segmentation network. The method comprises the following steps: acquiring an asphalt road crack data image by using a shooting device; preprocessing road crack data information; inputting the preprocessed data into a trained semantic segmentation network to obtain crack state information; determining the severity of the crack according to the crack state information; and the semantic segmentation network comprises an input preprocessing module, an improved ASPP module, an attention mechanism module, an output convolution calculation module and an output post-processing module. The training of the semantic segmentation network comprises the following steps: acquiring crack data, marking the acquired data to obtain labeled road crack data; normalizing the labeled crack data; preprocessing the normalized data by using the input preprocessing module; inputting the preprocessed data into the improved ASPP module after the backbone network generates an effective feature layer to extract features; learning the correlation between channels by using the channel attention module to adaptively adjust the weight of the channels; adjusting the number of channels by using the output convolution calculation module after obtaining the final effective layer to obtain a segmentation result; converting the segmentation result into a probability value by using the output post-processing module; calculating a model loss function according to the probability value to adjust the model parameters, and the training of the model is completed when the loss function converges. The atrous convolution layers of the improved ASPP structure are organized in a cascaded manner, wherein the dilution rate of each layer is increased layer by layer, the output of each atrous layer is cascaded with the input feature map and all outputs from lower layers, and the cascaded feature map is fed to the next layer. The preprocessing process of the input preprocessing module on the data comprises the following steps: preprocessing the data collected by two cameras, image enhancement is performed on the collected data, the length of the collected data is H*W, and H*W tensor data is obtained. The ECA attention module is an efficient channel attention module, which only designs a few parameters but has obvious effect gain, appropriate cross-channel interaction can significantly reduce the complexity of the model while maintaining the performance, and the non-reduced local cross-channel interaction strategy effectively avoids the influence of dimension reduction on the learning effect of the channel attention. The loss function of the model comprises a cross-entropy loss function and a Dice Loss function; in binary classification, cross-entropy is commonly used to evaluate the difference between labels and predictions; the calculation is as shown in formula (2), wherein y represents a label, y' represents a prediction, and N represents the number of pixels; where d l denotes the dilation rate of the l-th layer, [···] denotes concatenation operation, [y l-1 ,y l-2 ,…,y0] denotes the feature map formed by concatenating the outputs of all previous layers; SP Bar- pooling is added on the basis of DenseASPP, which deploys a long strip-shaped pooling kernel shape along the spatial dimension, so it can capture the long-distance relationship of isolated regions, while keeping a narrow kernel shape in other spatial dimensions, which is convenient for capturing local context and preventing irrelevant areas from interfering with label prediction. 2.The road crack detection method based on the improved DeepLabv3+ model according to claim 1, wherein, The Dice Loss is named after the Dice coefficient, which is a measure function for evaluating the similarity of two samples, and the greater the value is, the more similar the two samples are; the calculation of the Dice coefficient is as shown in formula (3): 3.The road crack detection method based on the improved DeepLabv3+ model according to claim 1, wherein, Wherein X represents the true situation of the crack, Y represents the predicted situation, |X intersection Y| represents the number of foreground intersection pixels between X and Y, |X| and |Y| respectively represent the number of pixels of the foreground (crack) in the label and the prediction in a specific image; and the Dice Loss expression is as shown in formula (4): 4.The road crack detection method based on the improved DeepLabv3+ model according to claim 1, wherein, ​ ​ ​ L = L ce + L dice (5) As shown in formula (5), the final loss function takes the form of the sum of the cross-entropy function and the Dice loss function.

Images