A pavement crack segmentation method based on MA-Xnet

Abstract

The present invention discloses a pavement crack segmentation method based on MA‑Xnet, including step S1: obtaining a concrete crack image data set; step S2: constructing a MA‑Xnet network model; step S3: training the MA‑Xnet network model; step S4: crack segmentation effect test, using the trained MA‑Xnet network model to segment the samples in the test set and generate a segmentation result graph. The features of the encoders of two basic UNets are mapped to the decoder part of the other party to form an X-shaped network with dual input and output. Then, Xnet is used as the basic network, and two MobileNetV2 feature extractions are used as dual encoders. Different attention modules are added to the two decoder parts to model the semantic dependency relationships in the spatial dimension and the channel dimension respectively; then the outputs of the two attention modules are fused to generate the final result graph. While having a good segmentation speed, the accuracy of crack segmentation is further improved.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a road surface crack segmentation method based on MA-Xnet. Background Art

[0002] Real-time monitoring of pavement cracks is crucial to the successful maintenance of road infrastructure. With the continuous expansion of urbanization and the rapid development of highways, the surface area of ​​concrete roads is getting larger and larger. However, with the increase in construction time, frequent use of highways and unexpected disasters caused by bad weather, the pavement of roads will have different degrees of depressions and cracks. This leads to an increasing demand for maintaining these roads to keep them in a usable state. Effective crack detection methods can help establish appropriate road maintenance and repair strategies, save maintenance budgets, reduce traffic accidents, and protect people's lives.

[0003] At present, most of the existing crack detection and segmentation technologies rely on some machine learning algorithms, such as threshold-based methods, edge-based detection methods, region growing methods, and more advanced Medoidshift algorithms and TurboPixels, a fast superpixel generation algorithm based on geometric flow. However, these algorithms are generally susceptible to noise, lack robustness to complex backgrounds, and generate a lot of redundancy during the calculation process, which increases the complexity of the calculation.

[0004] With the development of deep learning since 2015, deep learning can overcome the shortcomings of machine learning algorithms, such as weak learning ability, limited generalization ability, and insufficient robustness in complex backgrounds. Deep learning algorithms have a high degree of generalization and learning ability, can accurately extract image features, and have high real-time performance and stronger expression ability. Summary of the invention

[0005] The present invention provides a pavement crack segmentation method based on MA-Xnet, aiming to solve the problems of existing semantic segmentation algorithms being insensitive to details, lacking spatial consistency, having too large parameters in the model, or lacking contextual information in image segmentation results, which lead to slow model training and poor segmentation accuracy in segmentation tasks.

[0006] The present invention is implemented as follows: a pavement crack segmentation method based on MA-Xnet, characterized in that it comprises the following steps:

[0007] Step S1: Obtain a concrete crack image dataset, and randomly divide all images in the concrete crack image dataset into a training set, a validation set, and a test set in proportion;

[0008] Step S2: constructing a MA-Xnet network model, wherein the MA-Xnet network is an X-type network with dual inputs and outputs composed of two Mobile-Unets, wherein the two Mobile-Unets map features by exchanging jump connections, and the MA-Xnet network has two parallel end-to-end encoder and decoder architectures;

[0009] Step S3: training the MA-Xnet network model, using samples in the training set and samples in the validation set to train the constructed MA-Xnet network, and using the Adam optimizer algorithm to iteratively update the network weights based on the training data until the loss value trend of the MA-Xnet network model converges;

[0010] Step S4: Crack segmentation effect test: Use the trained MA-Xnet network model to segment the samples in the test set and generate a segmentation result map.

[0011] Furthermore, in step S1, the concrete crack image dataset is randomly selected from concrete crack images collected from various campus buildings of Middle East Technical University.

[0012] Furthermore, in step S2, the specific steps of constructing the MA-Xnet network model include: mapping the features of the encoders of the two Mobile-Unets to the decoder parts of the other party to form an X-shaped network with dual input and output, and then using the X-shaped network as the basic network, using two MobileNetV2 feature extractions as dual encoders, adding different attention modules to the two decoder parts, respectively modeling the semantic dependency relationships in the spatial dimension and the channel dimension, and then fusing the outputs of the two attention modules to generate the final result graph.

[0013] Furthermore, in step S2, the MobileNetV2 adopts a depth-wise separable convolution with an inverted residual bottleneck, including two independent layers, the first layer is a channel-by-channel convolution, which performs lightweight filtering by applying a single convolution filter to each input channel; the second layer is a point-by-point convolution of 1×1 convolution, which constructs new features by calculating linear combinations of input channels.

[0014] Furthermore, in step S2, different attention modules are added to the two decoder parts, specifically a position attention module and a channel attention module.

[0015] Furthermore, the position attention module has a global context view and encodes context information into local features; the channel attention module shows the interdependence between modeling channels.

[0016] Furthermore, in step S3, during the training of MA-Xnet, a loss function and optimizer based on the Keras deep learning library are established, and a cross entropy function is used as the loss function. The loss function is:

[0017]

[0018] Among them, i represents the pixel in the image, n represents the total number of image pixels, and y i represents the expected output of pixel i; Represents the actual output of pixel i.

[0019] Furthermore, in step S2, the activation function used in the MobileNetV2 is a nonlinear activation function.

[0020] Furthermore, the nonlinear activation function is a ReLU6 function:

[0021] f (x) =min{max(0,x)6}

[0022] Among them, x is the input from the previous layer of neural network.

[0023] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention discloses a road crack segmentation method based on MA-Xnet, which integrates the advantages of Mobile-Unet and DANet and proposes an X-type network architecture to perform semantic segmentation on crack images. The features of the encoders of the two Unets are mapped to the decoder part of the other party to form an X-type network architecture. The encoder part of the network uses two parallel MobileNetV2s for feature extraction, which has excellent network learning time and segmentation speed. Different attention modules are added to the two parallel decoder parts to model the semantic dependency relationship in the spatial dimension and the channel dimension, and then the outputs of the two attention modules are fused to generate the final result graph. This makes MA-Xnet more useful for segmentation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a flow chart of the present invention;

[0025] Figure 2 It is the network structure diagram of MA-Xnet of the present invention;

[0026] Figure 3 It is a diagram showing the test set segmentation results obtained by the method of the present invention and the existing method. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0028] In the description of the present invention, it should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, in the description of the present invention, "multiple" means two or more, unless otherwise clearly and specifically defined.

[0029] Example

[0030] See also Figure 1-3 , the present invention provides a technical solution:

[0031] A pavement crack segmentation method based on MA-Xnet, characterized by comprising the following steps:

[0032] Step S1: Obtain a concrete crack image dataset, and randomly divide all images in the concrete crack image dataset into a training set, a validation set, and a test set in proportion;

[0033] Step S2: constructing a MA-Xnet network model, wherein the MA-Xnet network is an X-type network with dual inputs and outputs composed of two Mobile-Unets, wherein the two Mobile-Unets map features by exchanging jump connections, and the MA-Xnet network has two parallel end-to-end encoder and decoder architectures;

[0034] Step S3: training the MA-Xnet network model, using samples in the training set and samples in the validation set to train the constructed MA-Xnet network, and using the Adam optimizer algorithm to iteratively update the network weights based on the training data until the loss value trend of the MA-Xnet network model converges;

[0035] Step S4: Crack segmentation effect test: Use the trained MA-Xnet network model to segment the samples in the test set and generate a segmentation result map.

[0036] In step S1, the concrete crack image dataset is randomly selected from concrete crack images collected from various campus buildings of Middle East Technical University.

[0037] In step S2, the specific steps of constructing the MA-Xnet network model include: mapping the features of the encoders of the two Mobile-Unets to the decoder parts of the other party to form an X-shaped network with dual input and output, and then using the X-shaped network as the basic network, using two MobileNetV2 feature extractions as dual encoders, adding different attention modules to the two decoder parts, respectively modeling the semantic dependency relationships in the spatial dimension and the channel dimension, and then fusing the outputs of the two attention modules to generate the final result graph.

[0038] In step S2, the MobileNetV2 adopts depth-wise separable convolution with inverted residual bottleneck, including two independent layers. The first layer is channel-by-channel convolution, which performs lightweight filtering by applying a single convolution filter to each input channel; the second layer is a point-by-point convolution of 1×1 convolution, which constructs new features by calculating linear combinations of input channels.

[0039] In step S2, different attention modules are added to the two decoder parts, specifically a position attention module and a channel attention module.

[0040] The position attention module has a global context view and encodes context information into local features; the channel attention module explicitly models the interdependencies between channels.

[0041] In step S3, during the training of MA-Xnet, a loss function and optimizer based on the Keras deep learning library are established, and a cross entropy function is used as the loss function. The loss function is:

[0042]

[0043] Among them, i represents the pixel in the image, n represents the total number of image pixels, and y i represents the expected output of pixel i; Represents the actual output of pixel i.

[0044] In step S2, the activation function used in the MobileNetV2 is a nonlinear activation function.

[0045] The nonlinear activation function is the ReLU6 function:

[0046] f (x) =min{max(0,x)6}

[0047] Among them, x is the input from the previous layer of neural network.

[0048] Test example

[0049] A pavement crack segmentation method based on MA-Xnet, the method comprising the following steps:

[0050] Step 1: Randomly select 2,000 images (1,000 positive and negative samples) from the open source dataset of concrete crack images collected from various campus buildings of Middle East Technical University as the segmented dataset. Positive samples are with cracks, and negative samples are without cracks. And divide them into 1,200 training sets, 400 validation sets, and 400 test sets according to 60%:20%:20%.

[0051] Step 2: Build the MA-Xnet network: Map the features of the encoders of the two basic UNets to the decoder parts of each other to form an X-shaped network with dual inputs and outputs. Then use Xnet as the basic network and use two MobileNetV2 feature extractions as dual encoders. Different attention modules are added to the two decoder parts to model the semantic dependencies in the spatial dimension and channel dimension respectively. Then, the outputs of the two attention modules are fused to generate the final result graph.

[0052] like Figure 2 As shown in the figure, the MA-Xnet network constructed in step 2 is a parallel dual-input and dual-output X-type network that jumps and maps the features of its encoder part to the decoder part of the other party. During the downsampling process, the depth-separable convolution in MobileNetV2 is used instead of the original standard convolution as the feature extraction layer, and the inverted residual block is used, which greatly reduces the complexity of the network.

[0053] MobileNetV2 in step 2 uses depth-wise separable convolution, which consists of two independent layers. The first layer is called channel-by-channel convolution, which performs lightweight filtering by applying a single convolution filter to each input channel; the second layer is a 1×1 convolution, called point-by-point convolution, which is responsible for constructing new features by calculating linear combinations of input channels. For a traditional standard convolution layer, the input tensor formula is:

[0054] L i =h i × i ×d i

[0055] Among them, h i Indicates the height of the image, w i Indicates the width of the image, d i Represents the channel of the image. Use a size of k×k×d j After filtering, an output tensor is generated:

[0056] L i =h i ×i ×d j

[0057] Then the computational complexity of the traditional standard convolutional layer is:

[0058] Cost = h i ·w i ·d i ·d j ·k 2

[0059] The computational cost of convolution using depth separable is:

[0060] Cost = h i ·w i ·d i (k 2 +d j )

[0061] By comparison, it can be found that the computational complexity of depthwise separable convolution is about k times lower than that of traditional standard convolution. 2 times, k is the convolution kernel size.

[0062] MobileNetV2 in step 2 uses an inverted residual block. The structure of the inverted residual block is a method to reduce the amount of computation by replacing the common structure connected between feature maps with a structure with large channels between component bottlenecks. The number of channels at both ends of the inverted residual module is relatively small, belonging to a low-dimensional space. For low-dimensional space, linear mapping will preserve features, while nonlinear mapping will destroy features. Therefore, in MobileNetV2, we use a linear bottleneck to prevent nonlinear destruction of too much information.

[0063] The nonlinear activation function is the ReLU6 function:

[0064] f (x) =min{max(0,x)6}

[0065] Among them, x is the input amount from the previous layer of neural network entering the neuron.

[0066] Compared with the traditional Sigmoid activation function, ReLU6 can obtain sparse expression and has better performance than the traditional Sigmoid activation function. ReLU6 is a common ReLU with a maximum output value of 6, which gives it better numerical resolution.

[0067] In the upsampling stage, the features are still transferred upward layer by layer, but the jump connection part is no longer connected to its own upsampling layer, but jumps to the other upsampling layer. After that, the two decoder parts perform 4 deconvolutions, and then the features output by the 4th deconvolution layer are input into two attention modules, the position attention module and the channel attention module.

[0068] The position attention module described in step 2 has a global context view and can selectively aggregate context according to the spatial attention map. The position attention module encodes broader context information into local features, thereby enhancing its representation capability.

[0069] The channel attention module can explicitly model the interdependence between channels, and by utilizing the interdependence between channel mappings, it can emphasize interdependent feature mappings and improve the feature representation of specific semantics, which helps to improve the discriminability of features.

[0070] In step S3, during the training of MA-Xnet, a loss function and optimizer based on the Keras deep learning library are established, and the cross entropy function is used as the loss function. The loss function is:

[0071]

[0072] Among them, i represents the pixel in the image, n represents the total number of pixels in the image, and y i represents the expected output of pixel i; Represents the actual output of pixel i; Adam is used as the optimizer of network weights.

[0073] Specifically, the training parameters in step 3 are set as follows: the initial learning rate is 0.001, the weight decay is 5×10 -4 , momentum is 0.9, batchsize is 10, the number of epochs is 50, dropout is 0.5, and an early stopping mechanism is set to prevent overfitting. When the change value of the loss function val_loss on the validation set is less than 0.0001 after 20 epochs, training and validation stop.

[0074] Step 4, crack segmentation effect test: Use the MA-Xnet network trained in step 3 to segment the samples of the test set and generate a segmentation result graph. The segmentation method of the present invention is compared with that of FCN-8s, Mobile-Unet, and DANet. The segmentation result graph is shown in the figure below. Figure 3 shown.

[0075] From the segmentation results, we can see that compared with other networks:

[0076] FCN-8s performs poorly on all images because it does not consider the relationship between pixels and ignores the spatial rules used in common segmentation methods based on pixel classification, resulting in loss of local information and local refinement capabilities.

[0077] DANet, using dual attention modules to adaptively aggregate long-distance context information, gives more accurate segmentation results than FCN-8s. However, compared with Mobile-Unet and MA-Xnet, it can be found that the segmentation accuracy is not very high. For example, when segmenting the crack image No. 00002, there is an obvious fault at the bottom, and when segmenting the crack image No. 00005, the same problem occurs at the top. DANet has deeper layers and slower learning ability than Mobile-Unet and MA-Xnet. Therefore, when the number of samples is not large, DANet cannot learn enough features, resulting in low segmentation accuracy.

[0078] Mobile-Unet has a high detection rate and good application value, but the edges of the segmented images are not very smooth. For example, when segmenting the crack image numbered 00002, the result has obvious burrs. When segmenting the crack image numbered 00004, the bulge on the far left is not segmented as accurately as the proposed algorithm. In addition, when segmenting the crack image numbered 00005, there are also subtle defects in the middle part.

[0079] Compared with other methods, the MA-Xnet segmentation result map proposed in this paper is more accurate in details and closer to the actual segmentation label groundtruth map. While ensuring the accuracy, it also has a good segmentation speed, combining the advantages of Mobile-Unet and DANet.

[0080] Moreover, the performance comparison of each network is shown in Table 1:

[0081]

[0082] Table 1

[0083] From Table 1, we can see that MA-Xnet outperforms other methods in terms of Accuracy, MIoU, Recall, F1, and AP. Therefore, the segmentation performance of the proposed network is verified by comparing with other networks.

[0084] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A pavement crack segmentation method based on MA-Xnet, characterized in that: The following steps are involved: Step S1: Obtain a concrete crack image dataset, and randomly divide all images in the concrete crack image dataset into a training set, a validation set, and a test set in proportion; Step S2: constructing a MA-Xnet network model, wherein the MA-Xnet network is an X-type network with dual inputs and outputs composed of two Mobile-Unets, wherein the two Mobile-Unets map features by exchanging jump connections, and the MA-Xnet network has two parallel end-to-end encoder and decoder architectures; Step S3: training the MA-Xnet network model, using samples in the training set and samples in the validation set to train the constructed MA-Xnet network, and using the Adam optimizer algorithm to iteratively update the network weights based on the training data until the loss value trend of the MA-Xnet network model converges; Step S4: Crack segmentation effect test: Use the trained MA-Xnet network model to segment the samples in the test set and generate a segmentation result map.

2. The pavement crack segmentation method based on MA-Xnet according to claim 1, characterized in that: In step S1, the concrete crack image dataset is randomly selected from concrete crack images collected from various campus buildings of Middle East Technical University.

3. The pavement crack segmentation method based on MA-Xnet according to claim 1, characterized in that: In step S2, the specific steps of constructing the MA-Xnet network model include: mapping the features of the encoders of the two Mobile-Unets to the decoder parts of the other party to form an X-shaped network with dual input and output, and then using the X-shaped network as the basic network, using two MobileNetV2 feature extractions as dual encoders, adding different attention modules to the two decoder parts, respectively modeling the semantic dependency relationships in the spatial dimension and the channel dimension, and then fusing the outputs of the two attention modules to generate the final result graph.

4. The pavement crack segmentation method based on MA-Xnet according to claim 3 is characterized in that: In step S2, the MobileNetV2 adopts depth-wise separable convolution with inverted residual bottleneck, including two independent layers. The first layer is channel-by-channel convolution, which performs lightweight filtering by applying a single convolution filter to each input channel; the second layer is a point-by-point convolution of 1×1 convolution, which constructs new features by calculating linear combinations of input channels.

5. The pavement crack segmentation method based on MA-Xnet according to claim 3, characterized in that: In step S2, different attention modules are added to the two decoder parts, specifically a position attention module and a channel attention module.

6. The pavement crack segmentation method based on MA-Xnet according to claim 5, characterized in that: The position attention module has a global context view and encodes context information into local features; the channel attention module explicitly models the interdependencies between channels.

7. The pavement crack segmentation method based on MA-Xnet according to claim 1, characterized in that: In step S3, during the training of MA-Xnet, a loss function and optimizer based on the Keras deep learning library are established, and a cross entropy function is used as the loss function. The loss function is: Among them, i represents the pixel in the image, n represents the total number of pixels in the image, and y i represents the expected output of pixel i; Represents the actual output of pixel i.

8. The pavement crack segmentation method based on MA-Xnet according to claim 4, characterized in that: In step S2, the activation function used in the MobileNetV2 is a nonlinear activation function.

9. The pavement crack segmentation method based on MA-Xnet according to claim 8, characterized in that: The nonlinear activation function is the ReLU6 function: f (x) =min{max(0,x)6} Among them, x is the input from the previous layer of neural network.

Images