A method for detecting pavement cracks by improving a UNet network structure

Abstract

The present application relates to the technical field of pavement crack detection, and specifically relates to a pavement crack detection method with improved UNet network structure, which improves the existing UNet network structure, integrates multiple intermediate nodes, multi-scale skip connection structure, channel and spatial attention mechanism and other modules into the network, uses real pavement images for training and testing, obtains a network model and parameters meeting the requirements, and finally detects cracks in the image to be detected. The present application solves the following three technical problems: first, the crack image has a complex background and many noises, making it difficult to extract features; second, the crack detection algorithm only uses single-level features, which may lose details; third, the network ignores crack semantic features and detail feature information, causing performance degradation; ultimately, the present application improves the accuracy of crack detection and improves the defects of the prior art.

Description

Technical Field

[0001] This invention relates to the field of pavement crack detection technology, and specifically to a pavement crack detection method with an improved UNet network structure. Background Technology

[0002] Cracks are common defects on the surfaces of various physical structures. Road surface cracks not only affect the aesthetics of roads but also pose a threat to the safety of vehicles, impact the overall structure of the roadbed, significantly shorten the road's lifespan, and make the road surface more susceptible to further damage. Initially, crack detection relied on manual visual inspection, which was not only labor-intensive but also resulted in subjective findings. Deep learning-based crack detection algorithms, however, can avoid the subjectivity and safety issues of manual instrument inspection, making them a hot research topic in the industry.

[0003] Existing crack images suffer from drawbacks such as high noise levels, low contrast, and poor continuity. The shapes of crack images vary, and the road environment is complex, all of which place high demands on the detection efficiency and accuracy of crack detection algorithms.

[0004] Specifically, the existing technology has the following drawbacks:

[0005] (1) Low-level features extracted from images are rich in spatial information and contain many location details, but lack semantic information. In contrast, high-level features are rich in semantic information but lack spatial information and have poor perception of details. For crack images with complex backgrounds, it is difficult to achieve good results if only a single extracted feature is used.

[0006] (2) Many crack detection algorithms can only adapt to crack images under certain conditions, such as road surfaces, bridge surfaces, walls and lining joints, and are difficult to perform well in datasets with various complex backgrounds, lacking generalization ability.

[0007] (3) Crack detection images have complex backgrounds and contain a lot of noise, such as road surface textures, stains, and water seepage. Their image continuity is poor, contrast is low, and it is difficult to define crack features. Many image segmentation methods will have significantly reduced detection performance in such cases. Summary of the Invention

[0008] The purpose of this invention is to provide an improved road surface crack detection method using a UNet network structure. This method introduces and improves the UNet convolutional neural network by incorporating multiple intermediate nodes, multi-scale skip connection structures, and channel and spatial attention mechanisms into the network for crack detection, thereby improving the accuracy of crack detection.

[0009] To achieve the above objectives, this invention provides an improved method for detecting road surface cracks using a UNet network structure, comprising the following steps:

[0010] Step 1: Use an image acquisition device to obtain a real road surface image;

[0011] Step 2: After processing a selection of real road surface images, construct training and testing sets;

[0012] Step 3: Improve the UNet network structure and train the improved UNet network model using the training set;

[0013] Step 4: Use the test set to test the trained network model, and obtain the test data to evaluate the network model performance using three metrics: Precision, Recall, and F-score.

[0014] Step 5: If the indicators meet the requirements, it proves that the network model has met the requirements, and the current parameters and model are retained; otherwise, modify the network parameters and repeat steps 3 and 4.

[0015] Step 6: Use a network model and parameters that meet the requirements to perform crack detection on the image to be detected.

[0016] Optionally, during the process of constructing the training and test sets after processing some real road surface images, images containing road surface cracks in the real road surface images are retained, cropped to an appropriate size, and labeled to obtain the original image and the labeled grayscale image. The training and test sets are then constructed using the images.

[0017] Optionally, the process of improving the UNet network architecture includes the following steps:

[0018] By adding intermediate nodes to UNet, the network can be segmented into local U-shaped networks of different depths. Side outputs are obtained from each local network, and all side outputs are spliced ​​and fused to obtain the final prediction result. At the same time, a deep supervision network is constructed.

[0019] Improve the skip connection structure in UNet by using a multi-scale skip connection method to obtain full-scale feature information;

[0020] In U-shaped networks with depths of 2 and 3, add spatial attention modules to full-resolution nodes; in U-shaped networks with depths of 4 and 5, add channel attention modules to full-resolution nodes.

[0021] Each convolutional block in the network contains a 3×3 convolutional layer, a batch normalization layer, and two activation layers each. Max pooling is used for downsampling.

[0022] Optionally, intermediate nodes are added to the UNet to split the network into local U-shaped networks of different depths and construct a deep supervision network. Specifically, N - n - 1 intermediate nodes are added between the encoding layer and the decoding layer nodes of the nth layer of the UNet, where N is the total depth of the network and 1 < n < 4. Each intermediate node contains a 3×3 convolutional layer, a batch normalization layer, and an activation layer twice. On the full-resolution nodes of the local U-shaped networks at each depth, 1×1 convolution and bilinear interpolation are used to obtain a feature map with the same resolution as the original input image, and then all the feature maps are fused to obtain the final fused output result. Depth supervision is performed on the side outputs to provide guidance for the final output result.

[0023] Optionally, the spatial attention module processes the spatial dimension of the input feature map. The feature map with an input of C×H×W is respectively subjected to max pooling and average pooling to obtain two feature maps of 1×H×W. After merging these two feature maps, a feature map of 2×H×W is obtained. After processing with a 7×7 convolutional kernel and then activation by the Sigmoid function, weight information at the spatial level is obtained, and the result is obtained after fusion with the original input feature map.

[0024] The channel attention module processes the channel dimension of the input feature map. After average pooling of the feature map with an input of C×H×W, the spatial dimension is compressed to obtain a feature map of C×1×1. After linear transformation and activation processing of this feature map, weight information at the channel level is obtained, and the result is obtained after fusion with the original input feature map.

[0025] Here, C is the number of channels of the feature map, H is the height, and W is the width.

[0026] Optionally, the multi-scale skip connection method is that each decoder layer node in the network is respectively connected to the encoder nodes of the same layer and shallower layers as the decoder, the intermediate nodes of the same layer, and the decoder layer nodes of deeper layers.

[0027] Optionally, the process of using the multi-scale skip connection method to obtain full-scale feature information is as follows: specifically, when the connection node is at a deeper level, its feature map needs to be upsampled to increase its size to the same as that of the decoder layer node; when the connection node is at a shallower level, its feature map needs to be downsampled, and its dimension is reduced to the same as that of the decoder layer node through max pooling operation; after the above operations, all the feature maps with the same size as the decoder node are concatenated.

[0028] This invention provides an improved road surface crack detection method using a UNet network structure. The existing UNet network structure is improved by incorporating multiple intermediate nodes, a multi-scale skip connection structure, and channel and spatial attention mechanisms. The method is trained and tested using real road surface images to obtain a network model and parameters that meet the requirements. Finally, crack detection is performed on the images to be detected. This invention solves three technical problems: first, the complexity and noise of crack images make feature extraction difficult; second, crack detection algorithms often use only single-level features, potentially losing details; and third, the network neglects semantic and detailed crack features, leading to performance degradation. Ultimately, this invention improves the accuracy of crack detection and overcomes the shortcomings of existing technologies. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a schematic flowchart of a road surface crack detection method based on an improved UNet network structure according to the present invention.

[0031] Figure 2 This is a schematic diagram of the feature map construction of the third decoding layer in the existing UNet3+ network.

[0032] Figure 3 This is a structural diagram of the classification guidance module in the existing UNet3+ network.

[0033] Figure 4 This is a schematic diagram of the existing UNet3+ network structure.

[0034] Figure 5 This is a partial U-shaped network structure diagram of the improved UNet network of this invention.

[0035] Figure 6 This is a structural diagram of the channel attention module and spatial attention module of the improved UNet network of this invention.

[0036] Figure 7 This is a schematic diagram of the structure of the improved UNet network of this invention.

[0037] Figure 8 This is a comparison image of the test results of specific embodiments of the present invention and other algorithms on the DeepCrack, CFD, and CRACK500 datasets. Detailed Implementation

[0038] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0039] This invention provides an improved method for detecting road surface cracks using a UNet network structure, comprising the following steps:

[0040] S1: Use image acquisition equipment to obtain real road surface images;

[0041] S2: Select a portion of real road surface images, process them, and construct the training and test sets;

[0042] S3: Improve the UNet network structure and train the improved UNet network model using the training set;

[0043] S4: Use the test set to test the trained network model, and obtain test data to obtain three metrics: Precision, Recall, and F-score to evaluate the network model performance.

[0044] S5: If the indicators meet the requirements, it proves that the network model has met the requirements, and the current parameters and model are retained; otherwise, after modifying the network parameters, repeat steps S3 and S4.

[0045] S6: Use a network model and parameters that meet the requirements to perform crack detection on the image to be detected.

[0046] UNet is a type of convolutional neural network, belonging to deep convolutional neural networks. The network consists of an encoder-decoder structure and skip connections, and to some extent addresses the problems of existing crack detection methods. This invention introduces and improves this network, enhancing the accuracy of crack detection. The specific process is as follows: Figure 1 As shown.

[0047] The present invention will be further described below in conjunction with existing technology and execution steps:

[0048] 1. Technical solutions of existing technologies

[0049] The existing UNet3+ network consists of skip connections, multi-scale feature fusion, and classification guidance modules. These modules can be used for feature extraction, feature fusion, and reducing over-segmentation, thereby improving the network's performance.

[0050] (1) Jump connection

[0051] Skip connections first appeared in ResNet to address the vanishing and exploding gradient problems that can occur with excessively deep networks. In this network, skip connections are used to interconnect the encoder and decoder, as well as to establish internal connections between decoder subnetworks. The feature map construction process of the third-layer decoder nodes is as follows: Figure 2 As shown, the decoder node directly receives feature maps from encoder layer nodes at the same level, and also receives feature maps from higher-level encoder layer nodes after dimensionality reduction via max pooling and lower-level decoder layer nodes after dimensionality upsampling. This structure allows each decoder layer node to combine smaller, same-scale feature maps from the encoder with larger-scale feature maps from the decoder, thus fully capturing both fine-grained detail information and coarse-grained semantic information.

[0052] (2) Multi-scale feature fusion

[0053] In the network, low-level nodes possess more detailed information but have higher image noise, while high-level nodes, after multiple convolutional operations for feature processing, have larger receptive fields and can capture higher-level semantic information. The multi-scale feature fusion module fuses feature maps from different levels, resulting in an image that possesses both low-level detailed information and high-level semantic information. The decoder layer nodes in the network have different sizes and require deconvolution processing, i.e., upsampling using 1×1 convolutions to ensure consistency in the feature maps of all decoder nodes. After deconvolution to unify the dimension of the feature maps, a 3×3 convolution kernel is used for convolution, followed by bilinear interpolation and activation processing to obtain the side outputs. All side outputs are concatenated and fused to obtain the final prediction result, giving it multi-level feature information and making it closer to the real crack image.

[0054] (3) Category Guidance Module

[0055] During image segmentation, background noise may remain in shallow layers, leading to oversegmentation and causing the network to mistakenly identify negative samples as positive ones. To achieve more accurate segmentation, a classification guidance module is added to the network, as shown in the diagram below. Figure 3 As shown, the feature map of the fifth-layer encoder node is processed by Dropout, convolution, max pooling, and activation to generate a two-dimensional tensor representing the probability of whether an image is a positive sample. Using the Argmax function, this tensor is converted into a single output of 0 or 1. Multiplying this output by the side outputs converted from the encoder node yields a more accurate classification result.

[0056] (4) UNet3+ network model

[0057] UNet3+ has an encoding-decoding structure, such as Figure 4As shown in the figure. In the encoding stage, the feature map goes through four downsampling stages, each of which contains a convolutional layer, a max pooling layer, and an activation layer. The decoder nodes are interconnected with the encoder nodes at higher levels and the same level, as well as the decoder nodes at lower levels through skip connection structures to obtain multi-scale feature information. After the size of the feature map is unified by transposed convolution for each decoder node, the side output is pulled, and it is combined with the classification guidance module. The side outputs processed by the classification guidance module are stitched and fused to obtain the final predicted output result.

[0058] However, after conducting experiments on three crack datasets, namely DeepCrack, CFD, and Crack500, using the UNet3+ network, there are a lot of noises in the resulting images, the crack image part is not obvious enough, and the details are not prominent enough.

[0059] 2. Improvements and innovations of the present invention

[0060] Furthermore, the present invention uses an improved UNet network as the backbone network, integrates multiple intermediate nodes, multi-scale skip connection structures, and modules such as channel and spatial attention mechanisms into the network, and constructs a pixel-level crack detection network.

[0061] (1) Intermediate nodes

[0062] Add N - n - 1 (where N is the total depth of the network, 1 < n < 4) intermediate nodes between the encoding layer and the decoding layer nodes of the nth layer of the UNet network. Each intermediate node contains a 3×3 convolutional layer, a batch normalization layer, and an activation layer twice. The existence of intermediate nodes enables the network to be divided into small blocks of different depths, and each small block can be regarded as a local encoder-decoder structure, as Figure 5 shown. In each local U-shaped network, the last layer node of the encoder has full-resolution feature information. The side output is obtained using a 1×1 convolution from all full-resolution nodes, and all side outputs are stitched and fused to obtain the final predicted result.

[0063] (2) Channel and spatial attention mechanism module

[0064] The attention mechanism module can suppress the attention to non-key areas and increase the weight of the attention areas. In the crack detection network, the attention mechanism module can adapt to the complex background of the road surface and the variability of the crack topology structure, and improve the accuracy of crack detection. The structural diagrams of the spatial attention mechanism module and the channel attention mechanism module are as Figure 6As shown. Spatial attention mechanisms focus on extracting spatial features, making the network more attentive to task-related regions. First, max pooling and average pooling are performed on the input feature map to obtain two feature maps of the same size. These two feature maps are concatenated and then convolved and activated to obtain a feature map with spatial weight information. This feature map is then fused with the original input image to obtain the final result. The dimensional parameters in the input image mainly include image size and the number of channels. Spatial attention mechanisms primarily compress the image channels to obtain spatial weight information, while channel attention mechanisms primarily compress the image size to obtain channel-level weight information. First, average pooling is performed on the input feature map to compress the spatial dimension to 1×1, while the number of image channels remains unchanged. Then, linear transformation and activation are performed on the feature map to obtain a feature map with channel-level weight information. This feature map is then fused with the original input image to obtain the final result.

[0065] (3) Multi-scale jump connection structure

[0066] In the UNet network, there is a skip connection structure between encoder and decoder nodes. However, this structure only exists between nodes within the same layer and lacks the ability to acquire multi-scale feature information. Nodes at different levels in the network possess different feature information. Lower-level feature maps have higher resolution and can extract more detailed information, but they have less semantic information and a smaller receptive field. Higher-level feature maps undergo multiple feature extraction operations, resulting in rich semantic information, but their image resolution is lower. This invention uses a multi-scale skip connection structure in the network to enable nodes to acquire feature information at different scales, improving the model's crack detection capability. Each decoder layer node in the network is connected to encoder nodes at the same and shallower layers, intermediate nodes at the same layer, and deeper decoder layer nodes. Specifically, when the connection node is at a deeper level, its feature map needs to be upsampled, increasing its size to the same dimension as the decoder layer node; when the connection node is at a shallower level, its feature map needs to be downsampled, undergoing max pooling to reduce its size to the same dimension as the decoder layer node. After these operations, all feature maps with the same size as the decoder node are concatenated. Decoder layer nodes processed by multi-scale skip connection can obtain feature information from various levels, resulting in richer feature information.

[0067] (4) Crack Detection Network Model

[0068] This invention adds intermediate nodes to UNet, enabling the network to be segmented into local U-shaped networks of different depths, and constructs a deep supervised network; it improves the skip connection structure in UNet by using a multi-scale skip connection method to obtain full-scale feature information; it adds spatial attention modules to the full-resolution nodes in U-shaped networks with 2 and 3 layers, and channel attention modules to the full-resolution nodes in U-shaped networks with 4 and 5 layers; each convolutional block in the network contains a 3×3 convolutional layer, a batch normalization layer, and two activation layers, and uses max pooling for downsampling. The network structure diagram proposed in this invention is shown below. Figure 7 As shown, the input image is first preprocessed, and the input feature map is fed into the network. This feature map undergoes five downsampling operations for feature extraction, each downsampling operation consisting of a convolutional block and a max pooling operation. Simultaneously, the feature information of each encoding layer node and intermediate nodes is fed into all nodes deeper than them, with each intermediate node containing a convolutional block. After passing through the encoding layer, the feature map is fed into the decoding layer, undergoing five upsampling operations, each upsampling operation consisting of a convolutional block and a bilinear interpolation operation. Finally, the feature map, after having its attention weights extracted by either the channel attention module or the spatial attention module, is subjected to a 1×1 convolution and a bilinear interpolation to extract the side outputs. All side outputs are then fused into the final predicted output.

[0069] Optionally, to verify the effectiveness of the proposed crack detection method, experiments were conducted on three crack datasets: DeepCrack, CFD, and Crack500. A portion of images from the DeepCrack dataset was used as the training set, while the remaining images from DeepCrack, along with those from the CFD and Crack500 datasets, served as the validation set. Furthermore, the proposed method was compared with image segmentation methods SegNet and UNet, and crack detection methods DeepCrackZ, DeepCrack, and FPHBN. The experimental results were evaluated using three metrics: Precision, Recall, and F-score. Partial detection results on the three datasets are shown below. Figure 8As shown, the images from left to right are (a) a real road surface image, (b) a labeled image, (c) the experimental results of SegNet, (d) the experimental results of DeepCrackZ, (e) the experimental results of DeepCrack, (f) the experimental results of FPHBN, (g) the experimental results of UNet, and (h) the experimental results of the network proposed in this invention. From top to bottom, they represent the experimental results on the DeepCrack dataset, the CFD dataset, and the Crack500 dataset, respectively. From a visual perspective, the crack detection images produced by the method proposed in this invention have clear and complete crack edges, less noise, and better detection performance. The experimental data on the three datasets are shown in Tables 1, 2, and 3. As can be seen from the tables, the three indicators of the method proposed in this invention are all higher than those of other methods. In summary, the crack detection method proposed in this invention is the best among all methods and has better crack detection performance.

[0070] Table 1 shows the experimental data on the DeepCrack dataset.

[0071]

[0072]

[0073] Table 2 shows the experimental data on the CFD dataset.

[0074] Methods Precision Recall F1 score SegNet 0.7771 0.7741 0.7756 DeepCrackZ 0.8271 0.8377 0.8324 DeepCrack 0.8333 0.8491 0.8412 FPHBN 0.8080 0.8133 0.8106 UNet 0.7201 0.7432 0.7315 Ours 0.8580 0.8565 0.8572

[0075] Table 3 shows the experimental data on the Crack500 dataset.

[0076] Methods Precision Recall F1 score SegNet 0.7720 0.7687 0.7704 DeepCrackZ 0.8208 0.8251 0.8229 DeepCrack 0.8273 0.8356 0.8314 FPHBN 0.7983 0.8081 0.8032 UNet 0.6946 0.6795 0.6870 Ours 0.8418 0.8435 0.8426

[0077] The above description discloses only one preferred embodiment of the present invention, and should not be construed as limiting the scope of the present invention. Those skilled in the art will understand that all or part of the processes of the above embodiments can be implemented, and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. A method for detecting pavement cracks using an improved UNet network structure, characterized in that, It includes the following steps: Step 1: Use an image acquisition device to obtain real road surface images; Step 2: Screen some real road surface images, process them, and construct a training set and a test set; Step 3: Improve the UNet network structure, and use the training set to train the improved UNet network model; The process of improving the UNet network structure includes the following steps: Add intermediate nodes in UNet to divide the network into local U-shaped networks with different depths, obtain side outputs from each local network, splice and fuse all side outputs to get the final prediction result, and at the same time construct a deep supervision network; Improve the skip connection structure in UNet and use a multi-scale skip connection method to obtain full-scale feature information; Add a spatial attention module to the full-resolution nodes in the U-shaped networks with 2 and 3 layers of depth, and add a channel attention module to the full-resolution nodes in the U-shaped networks with 4 and 5 layers of depth; Each convolutional block in the network contains a 3×3 convolutional layer, a batch normalization layer, and an activation layer twice, and max pooling is used for downsampling; The process of adding intermediate nodes in UNet to divide the network into local U-shaped networks with different depths and constructing a deep supervision network is specifically to add N-n-1 intermediate nodes between the encoding layer and the decoding layer nodes of the nth layer of the UNet network, where N is the total depth of the network, 1 < n < 4, and each intermediate node contains a 3×3 convolutional layer, a batch normalization layer, and an activation layer twice; on the full-resolution nodes of the local U-shaped networks with each depth, use 1×1 convolution and bilinear interpolation to process to obtain a feature map with the same resolution as the original input image, and then fuse all feature maps to get the final fused output result; Performing deep supervision on the side outputs can provide guidance for the final output result; Step 4: Use the test set to test the trained network model, obtain test data, and get three indicators of Precision, Recall, and F-score to evaluate the performance of the network model; Step 5: If the indicators meet the requirements, it proves that the network model meets the required requirements, and retain the current parameters and model; otherwise, modify the network parameters and repeat steps 3 and 4; Step 6: Use the network model and parameters that meet the requirements to detect cracks in the image to be detected.

2. The road surface crack detection method with an improved UNet network structure as described in claim 1, wherein In the process of screening some real road surface images, processing them, and constructing a training set and a test set, retain the images containing road surface cracks in the real road surface images, crop them into appropriate sizes, and perform annotation to obtain the original images and grayscale images with annotations, and then use the above images to construct a training set and a test set.

3. The road surface crack detection method with an improved UNet network structure as described in claim 2, wherein The spatial attention module processes the spatial dimension of the input feature map. It performs max pooling and average pooling on a feature map with input C×H×W to obtain two 1×H×W feature maps. These two feature maps are merged to obtain a 2×H×W feature map. After processing with a 7×7 convolution kernel and activation by the Sigmoid function, spatial weight information is obtained. This information is then fused with the original input feature map to obtain the result. The channel attention module processes the channel dimension of the input feature map. After average pooling, the spatial dimension of a feature map with input C×H×W is compressed to obtain a C×1×1 feature map. The feature map is then subjected to linear transformation and activation processing to obtain channel-level weight information, which is then fused with the original input feature map to obtain the result. Where C is the number of feature map channels, H is the height, and W is the width.

4. The road surface crack detection method with improved UNet network structure as described in claim 3, characterized in that, The multi-scale jump connection method involves each decoder layer node in the network being connected to encoder nodes at the same and shallower layers as the decoder, intermediate nodes at the same layer, and deeper decoder layer nodes.

5. The pavement crack detection method with improved UNet network structure as described in claim 4, characterized in that, The process of obtaining full-scale feature information using multi-scale skip connections involves the following steps: when the connection node is deeper, its feature map needs to be upsampled to increase its size to the same dimension as the decoder layer node; when the connection node is shallower, its feature map needs to be downsampled and reduced to the same dimension as the decoder layer node through max pooling; after the above operations, all feature maps with the same size as the decoder node are concatenated.

Images