Improved semantic segmentation network construction method and system for bridge and road crack identification

Abstract

The present invention belongs to the technical field of bridge and highway crack identification, and discloses an improved semantic segmentation network construction method and system for bridge and road crack identification. The method improves the convolution block, downsampling layer and upsampling layer in the UNet network, introduces a new spatial-channel attention mechanism, and retains the skip connection and U-shaped structure in the UNet network to construct an improved semantic segmentation network for crack identification. The present invention designs a highly specialized crack semantic segmentation network model with stronger generalization ability and higher accuracy. On the CRACK500, SDNET2018, and CFD public data sets, the IoU is 73.3%, and the F1 reaches 82.6%, which is much higher than other models. It is suitable for detecting cracks on bridge pavement, bridge towers and other bridge structures.

Description

Technical Field

[0001] The present invention belongs to the technical field of bridge and road crack identification, and in particular relates to an improved semantic segmentation network construction method and system for bridge and road crack identification. Background Art

[0002] With the continuous development of transportation, the maintenance mileage of bridges and roads continues to increase. After opening to traffic, due to the effects of vehicle loads and various natural environmental conditions, bridges and roads will suffer from various damages and diseases, resulting in a decline in their quality. Therefore, disease detection is an important means of structural maintenance for bridges and roads. Among the many damages, crack detection is often the most difficult. Cracks are early signs of serious structural damage and are also an important indicator for structural health assessment and detection. Traditional crack detection usually relies on manual labor, but there are a series of disadvantages such as heavy labor, long time consumption, and traffic impact. There are also some image-based automated detection methods and laser ground penetrating radar monitoring, but they also have low accuracy and complex image backgrounds, making it difficult to identify small cracks with blurred edges.

[0003] With the continuous development of deep learning and the improvement of computer computing power, a number of neural networks for crack detection have gradually emerged. In 2015, the FCN fully convolutional neural network came out, which replaced the fully connected layer in CNN with a convolutional layer, so that it can freely accept input images and output segmented images. In the same year, U-Net was particularly favored in the field of medical image segmentation by its U-shaped structure and the introduction of jump connections. In order to improve the recognition accuracy of the network, a series of U-Net variants came into being. ResUnet improved the recognition accuracy by introducing a residual structure. Unet++ reduced the loss in the feature transmission process by increasing the jump connection between the encoder and the decoder. Attation-UNet added a channel attention mechanism in the decoder downsampling and convolutional layer, so that the network focuses on In the connected areas of cracks, image noise is reduced. In 2017, the Deeplabv2 network was proposed. This network introduced void convolution, which increased the receptive field, can effectively extract multi-scale information, and improve the robustness of segmentation. Deepplabv3 is an improved version of Deeplabv2. It introduced series and parallel ASPP models, and introduced Xception to replace the convolution layer to improve the ability to extract multi-scale features. These classic networks and their variants have been successfully applied to crack detection, greatly reducing the time required for manual detection. However, cracks are often in the shape of small strips and have tiny edges. Although these networks have been successfully applied, the recognition accuracy of cracks with complex structures is not high, and manual secondary inspection is often required. Therefore, there are technical difficulties in designing a robust semantic segmentation network.

[0004] Through the above analysis, the problems and defects of the existing technology are as follows: in the existing bridge and highway crack image recognition technology, the accuracy of bridge and highway crack segmentation is low and the recognition effect is poor. Summary of the invention

[0005] In order to overcome the problems existing in the related art, the disclosed embodiments of the present invention provide an improved semantic segmentation network construction method and system for bridge and road crack identification, specifically relating to a new semantic segmentation network for identifying bridge and road cracks.

[0006] The technical solution is as follows: an improved semantic segmentation network construction method for bridge and road crack recognition, by improving the convolutional blocks, downsampling layers and upsampling layers in the UNet network, introducing a spatial-channel attention mechanism, retaining the skip connection and U-shaped structure in the UNet network, constructing an improved semantic segmentation network for crack recognition; specifically comprising the following steps:

[0007] S101, PConv lightweight convolution reconstruction convolution layer: replace the second convolution operation used for feature extraction in the two-dimensional convolution of U-Net with PConv lightweight convolution to improve spatial features. By controlling variables, the UNet network operation speed is improved only when the convolution layer is changed;

[0008] S102, introduces SCSA spatial-channel attention mechanism: Based on PConv lightweight convolution, the channel attention mechanism is added to the downsampling, upsampling and skip connection of the U-Net model to combine the image feature information between different levels and improve the attention to the cracks.

[0009] S103, reconstruction of downsampling layer: In the crack segmentation process, the feature information area of ​​the crack edge texture is small, so the SCSA space-channel attention mechanism is introduced, and the HWD wavelet downsampling layer is used to replace the maximum pooling layer to improve the U-Net network's retention of crack edge and fine line features;

[0010] S104, reconstruct upsampling layer: introduce SCSA space-channel attention mechanism, use the new upsampler DySample to reconstruct the upsampling layer, build an improved semantic segmentation network, complete image segmentation based on the improved semantic segmentation network, and realize crack structure recovery.

[0011] In step S101, PConv lightweight convolution reconstructs the convolution layer, including:

[0012] (1) Divide the input feature map into 32 channels, extract 8 channels and perform convolution with the convolution kernel, keep the remaining 24 channels unchanged, and output the output feature image of 8 channels;

[0013] (2) The 8-channel feature map obtained by the convolution operation is concatenated with the input feature map of the remaining 24 channels to obtain a complete convolution operation.

[0014] In step S102, the SCSA space-channel attention mechanism is introduced, including:

[0015] In the first step, the input feature map of size B×C×H×W is decomposed along the height and width dimensions using 1D convolution, and global average pooling is applied to each dimension. The feature map is decomposed through the Spilt operation to obtain four sub-features in two different directions.

[0016] In the second step, the four sub-features in two different directions are input into one-dimensional convolutions with convolution kernels of 3, 5, 7, and 9, respectively, and convolution operations of different scales are performed to extract information of different scales;

[0017] In the third step, the multi-scale features extracted in two different directions are spliced ​​separately, and then normalized within the group. After the sigmoid activation function, the two different direction features are fused, and the input features and output features are fused in combination with the residual structure, thereby completing the construction of the SMSA module;

[0018] The fourth step is to perform global average pooling on the feature map output by the SMSA module, and then perform a group normalization operation to ensure the independence of each sub-feature;

[0019] In the fifth step, the normalized feature map is convolved with 1×1 convolution kernels of different dimensions to obtain V, K, and Q feature images with different numbers of channels. V, K, and Q are respectively input into the self-attention mechanism for feature fusion. After obtaining the feature image, global average pooling and sigmiod function are performed and fused with the input feature map to output the final feature map and complete the SCSA space-channel attention mechanism.

[0020] In step S103, reconstructing the downsampling layer includes:

[0021] Step 1: The input feature map passes through the Haar wavelet transform module HWT to reduce the resolution of the feature map while retaining all the information of the feature map. The Haar wavelet transform generates subdivision components in four directions by decomposing the high-frequency and low-frequency information of the image, thereby quadrupling the number of channels and encoding part of the information from the spatial dimension to the channel dimension to obtain the feature map of the wavelet transform;

[0022] In the second step, the wavelet transformed feature map is passed into the feature learning module, which consists of a convolutional layer, an activation function, and batch normalization to complete the downsampling of the input image size.

[0023] In step S104, reconstructing the upsampling layer includes:

[0024] (a) Input a feature map of size H×W×C, and generate a dynamic weight factor S to output a feature map of size sH×sW×sC. The sampling set of the feature map is determined by the weight factor S. Using the sampling point generator, the input features are resampled by the grid sampling function to ensure that the feature map is dynamic, and then spliced ​​with the original feature map to output a feature map of size sH×sW×C, realizing DySample upsampling;

[0025] (b) The upsampler provides two methods to generate dynamic weight factor S and outputs weight factor S.

[0026] In step (b), two methods for generating a dynamic weight factor S include:

[0027] The first one is to input a feature map of size H×W×C to the linear connection layer, output a feature map of size H×W×2gS2, and reduce it to 0.25 times of the original size. The reduced O feature map is used as a bias and concatenated with the g feature map to output the weight factor S.

[0028] The second method is to reduce one of the two feature maps of size H×W×2gS2 to 0.5σ of the original size and concatenate it with the other feature map of size H×W×2gS2, and then perform bias to generate the O feature map, which is concatenated with the g feature map to generate the weight factor S.

[0029] The method further includes:

[0030] (1) The crack image is input and passed through the DoubleConv convolution layer to extract feature information. DoubleConv consists of a two-dimensional convolution and a Pconv, as well as an activation function and a BN layer;

[0031] (2) The feature map output by the DoubleConv convolution layer is input into the SCSA attention mechanism to retain the crack feature information. Part of the output feature layer is input into the HWD for downsampling, and part is retained and input into the decoder of the corresponding layer through the jump connection;

[0032] (3) After the feature map is downsampled by HWD wavelet, the number of channels doubles and the size is reduced to half of the original size. DoubleConv, SCSA, and HWD together constitute each layer of the encoder. After that, the input and output of the feature map are the same as the above steps. After passing through four layers of encoders, the feature extraction of image information is completed;

[0033] (4) The feature map after information extraction has the characteristics of small size and high number of channels. After DySample, the size of the feature map is doubled, and the number of channels remains unchanged;

[0034] (5) The feature map output by DySample passes through a 1×1 convolution layer to complete the compression of the number of channels. The output feature map is twice the size of the original feature map and half the number of channels.

[0035] (6) The feature map output by the 1×1 convolutional layer will be input into the SCSA attention mechanism to extract the backbone information of the feature map. The size and number of channels of the feature map remain unchanged.

[0036] (7) The feature map output by SCSA is input into the DoubleConv convolution layer to complete feature extraction and channel number compression;

[0037] (8) The feature map output by the convolution layer of DoubleConv is concatenated with the feature map output by the encoder at the bottom layer. The size and number of channels of the two feature maps are consistent. After concatenation, the number of channels of the feature map is doubled and input into the decoder of the previous layer.

[0038] (9) The decoder of each layer consists of DySample, 1×1Conv, SCSA and DoubleConv. After four times of decoder size and channel number restoration, the image segmentation task is completed.

[0039] Another object of the present invention is to provide an improved semantic segmentation network construction system for bridge and road crack identification, the system implements an improved semantic segmentation network construction method for bridge and road crack identification, and the system specifically includes:

[0040] PConv lightweight convolution reconstruction convolution layer module is used to replace the second convolution operation used for feature extraction in the two-dimensional convolution of U-Net with PConv lightweight convolution, improve spatial features, and improve the UNet network operation speed by controlling variables while only changing the convolution layer;

[0041] The SCSA spatial-channel attention mechanism module is introduced to improve the attention to cracks by adding the channel attention mechanism to the downsampling, upsampling and skip connection of the U-Net model to combine the image feature information between different levels based on the PConv lightweight convolution to blur the edges of cracks.

[0042] The reconstructed downsampling layer module is used to address the small feature information area of ​​the crack edge texture during crack segmentation. Based on the introduction of the SCSA space-channel attention mechanism, the HWD wavelet downsampling layer is used to replace the maximum pooling layer to improve the U-Net network's retention of crack edge and fine line features;

[0043] The reconstructed upsampling layer module is used to reconstruct the upsampling layer based on the introduction of the SCSA space-channel attention mechanism and the use of a new upsampler DySample to complete the construction of an improved semantic segmentation network, complete image segmentation based on the improved semantic segmentation network, and realize crack structure recovery.

[0044] Furthermore, the system is mounted on a computer-readable storage medium, and the computer-readable storage medium stores a computer program, and the computer program can realize the above functions when executed by a processor.

[0045] Furthermore, the system is mounted on an information data processing terminal, and when the information data processing terminal is implemented on an electronic device, it provides a user input interface to implement the above functions.

[0046] In combination with all the above technical solutions, the beneficial effects of the present invention are as follows:

[0047] The present invention is based on the classic U-Net network, introduces lightweight convolution to replace ordinary two-dimensional convolution, and adds a new channel-space hybrid attention mechanism in the encoder, decoder and jump connection for feature guidance and coordination of the relationship between multiple semantic layers. The downsampling maximum pooling layer in the encoder is replaced by a wavelet transform downsampling module to reduce the pixels of the image while retaining the diversity of the feature map. Finally, the transposed convolution upsampling layer in the encoder is replaced by a new dynamic upsampler to restore the resolution of the feature map, thereby greatly improving the network's accuracy for bridge and highway crack segmentation.

[0048] The present invention analyzes the obsolescence of the overall architecture of the original UNet model, reconstructs the upsampling and downsampling structures of the model and introduces a new attention mechanism, thereby designing a highly specialized crack semantic segmentation network model with stronger generalization ability and higher accuracy. On the CRACK500, SDNET2018, and CFD public datasets, the IoU is 73.3% and the F1 reaches 82.6%, which are much higher than other models. It is suitable for detecting cracks on bridge pavements, bridge towers and other bridge structures.

[0049] The high-precision crack recognition model provided by the present invention can efficiently identify cracks in roads and bridges, thereby reducing maintenance costs, extending service life, improving the safety and reliability of infrastructure, and reducing economic losses and casualties caused by sudden structural failures. The present invention improves the accuracy and efficiency of crack detection by reconstructing the overall structure of the original model, and solves the problem of insufficient recognition ability of the prior art under complex backgrounds. As a solution to fill the technical gap, this technology has the potential to promote the formulation and improvement of industry testing standards. Complex model structures are generally believed to provide higher accuracy, but they also bring problems such as high computational cost and difficulty in deployment. This solution overcomes the model complexity bias by only improving its different structures while retaining the main architecture of the original model. The model introduces an attention mechanism and a new up and down sampling module so that the model can perform well in different environments, thereby overcoming the environmental adaptability bias. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings herein are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description, serve to explain the principles of the present disclosure;

[0051] Figure 1 is a diagram of an improved semantic segmentation network construction method for bridge and road crack identification provided by an embodiment of the present invention;

[0052] Figure 2 is a standard convolution principle diagram provided by an embodiment of the present invention;

[0053] Figure 3 is a schematic diagram of a depth-separable convolution provided by an embodiment of the present invention;

[0054] Figure 4 It is a PConv lightweight convolution principle diagram provided by an embodiment of the present invention;

[0055] Figure 5 It is a schematic diagram of the SCSA space-channel attention mechanism provided by an embodiment of the present invention;

[0056] Figure 6 The comparison of the effect of the U-Net network and the network with the SCSA space-channel attention mechanism added to the original image crack-attention experiment; (a) is the original image, (b) is the original Unet model, and (c) is the added attention mechanism diagram;

[0057] Figure 7 The comparison of the effect of the U-Net network and the network with the SCSA spatial-channel attention mechanism added to the original image crack two-attention experiment; where (a) is the original image, (b) is the original Unet model, and (c) is the added attention mechanism diagram;

[0058] Figure 8 The comparison of the three-attention experiment results of the U-Net network and the network with the SCSA spatial-channel attention mechanism added to the original image; (a) is the original image, (b) is the original Unet model, and (c) is the diagram with the attention mechanism added;

[0059] Fig. 9 The comparison of the effect of the U-Net network and the network with the SCSA spatial-channel attention mechanism added to the original image crack four-degree attention experiment; where (a) is the original image, (b) is the original Unet model, and (c) is the added attention mechanism diagram;

[0060] Fig.10 This is the schematic diagram of HWD wavelet downsampling;

[0061] Fig.11 The image is the crack edge texture feature retention image in the HWD downsampling experiment comparison after replacing the original image; (a) is the original image, (b) is the maximum pooling image, and (c) is the Harr wavelet downsampling image;

[0062] Fig.12 The image showing the texture feature preservation of the crack edge in the HWD downsampling experiment after replacing the original image. (a) is the original image, (b) is the maximum pooling image, and (c) is the Harr wavelet downsampling image.

[0063] Fig.13 The image showing the texture feature preservation of the crack edge in the three-replacement HWD downsampling experiment of the original image; (a) is the original image, (b) is the maximum pooling image, and (c) is the Harr wavelet downsampling image;

[0064] Fig.14 The image showing the texture feature preservation of the crack edge in the four-replacement HWD downsampling experiment of the original image; (a) is the original image, (b) is the maximum pooling image, and (c) is the Harr wavelet downsampling image;

[0065] Fig.15 The schematic diagram of the DySample lightweight upsampling layer provided by the present invention; wherein (a) is a diagram of a common dynamic upsampler, (b) is a diagram of a sampling point generator with a static range factor of DySample, and (c) is a diagram of a sampling point generator with a dynamic range factor of DySample;

[0066] Fig.16 The crack structure recovery image in the DySample upsampling experiment is shown in Figure 1. (a) is the original image, (b) is the transposed convolution image, and (c) is the DySample upsampling image.

[0067] Fig.17The crack structure recovery image in the DySample upsampling experiment comparison after replacing the original image 2; (a) is the original image, (b) is the transposed convolution image, and (c) is the DySample upsampling image;

[0068] Fig.18 The crack structure recovery image in the DySample upsampling experiment comparison after three replacements of the original image; (a) is the original image, (b) is the transposed convolution image, and (c) is the DySample upsampling image;

[0069] Fig.19 This is a schematic diagram of the improved semantic segmentation network construction process of the present invention;

[0070] Fig. 20 The invention is Figure 1 Comparison of crack segmentation images of labels, original UNet model, UNet++, Attation-UNet, and the improved PHSD-UNet model of the present invention; (a) is the original image, (b) is the label image, (c) is the UNet image, (d) is the SegNet image, (e) is the UNet++ image, and (f) is the PHSD-UNet image;

[0071] Fig.21 The invention Figure 2 The original label UNet model, UNet++, Attation-UNet, and the improved PHSD-UNet model of the present invention are compared in crack segmentation diagram; where (a) is the original image, (b) is the label diagram, (c) is the UNet diagram, (d) is the SegNet diagram, (e) is the UNet++ diagram, and (f) is the PHSD-UNet diagram;

[0072] Fig. 22 The invention Figure 3 Comparison of crack segmentation images of labels, original UNet model, UNet++, Attation-UNet, and the improved PHSD-UNet model of the present invention; where (a) is the original image, (b) is the label image, (c) is the UNet image, (d) is the SegNet image, (e) is the UNet++ image, and (f) is the PHSD-UNet image. DETAILED DESCRIPTION

[0073] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention, so the present invention is not limited by the specific implementation disclosed below.

[0074] The innovation of the present invention lies in: proposing an improved U-Net model, which improves the crack detection capability in image segmentation; improving the efficiency of spatial feature extraction by replacing the two-dimensional convolution of U-Net with PConv lightweight convolution; introducing the SCSA space-channel attention mechanism to strengthen the attention to the crack edges; using HWD wavelet downsampling to replace the maximum pooling layer to retain subtle features; and using DySample to reconstruct the upsampling layer, which significantly improves the network's computing speed and detail recovery capability.

[0075] Embodiment 1, as Figure 1 As shown in the figure, the improved semantic segmentation network construction method for bridge and road crack recognition improves the convolutional blocks, downsampling layers, and upsampling layers in the UNet network, introduces a new spatial-channel attention mechanism, and retains the skip connection and U-shaped structure in the UNet network to construct an improved semantic segmentation network for crack recognition;

[0076] The specific steps include:

[0077] S101, PConv lightweight convolution reconstruction convolution layer: replace the second convolution operation used for feature extraction in the two-dimensional convolution of U-Net with PConv lightweight convolution to improve spatial features. By controlling variables, the UNet network operation speed is improved only when the convolution layer is changed;

[0078] S102, introduces SCSA spatial-channel attention mechanism: Based on PConv lightweight convolution, the channel attention mechanism is added to the downsampling, upsampling and skip connection of the U-Net model to combine the image feature information between different levels and improve the attention to the cracks.

[0079] S103, reconstruction of downsampling layer: In the crack segmentation process, the feature information area of ​​the crack edge texture is small, so the SCSA space-channel attention mechanism is introduced, and the HWD wavelet downsampling layer is used to replace the maximum pooling layer to improve the U-Net network's retention of crack edge and fine line features;

[0080] S104, reconstruct upsampling layer: introduce SCSA space-channel attention mechanism, use the new upsampler DySample to reconstruct the upsampling layer, build an improved semantic segmentation network, complete image segmentation based on the improved semantic segmentation network, and realize crack structure recovery.

[0081] Example 2: The improved semantic segmentation network is described in detail below.

[0082] (1) U-Net network. The U-Net network structure mainly consists of two parts: the encoder and the decoder. The encoder consists of a convolutional layer and a downsampling layer. The convolutional layer is used for feature extraction. The downsampling layer changes the size of the input image through maximum pooling. The decoding layer consists of a convolutional layer and an upsampling layer, which is used to restore and reconstruct the features in the encoder so that the size of the input image is consistent with the size of the output image. U-Net innovatively introduces skip connections, which combine the feature maps of the same level of the encoder and decoder to avoid feature loss caused by excessive network depth. Skip connections can also help the network learn distinctive feature information, thereby improving the network's learning ability.

[0083] (2) Improved semantic segmentation network suitable for bridge and road cracks.

[0084] (2.1) PConv lightweight convolution reconstruction convolution layer.

[0085] The convolution layer of the traditional UNet network extracts feature information through two 3×3 convolution kernels. The first convolution operation is used to extract features and increase the dimension of image features. The second convolution operation is used to increase the network depth. The UNet network is a lightweight convolution network. Although it can adaptively change the size of the convolution kernel to extract complex background features by referring to variable convolution, it will seriously hinder the network's computing speed.

[0086] Standard convolution ( Figure 2 ) Generate feature maps by convolution operations with different convolution kernels. The number of convolution kernels is consistent with the number of channels of the output feature map. In order to better extract features, the number of convolution kernels of standard convolution is large, which also leads to excessive convolution operation parameters. Therefore, depth-separable convolution (see Figure 3 ) was proposed. The depthwise convolution uses one convolution kernel to be responsible for one channel, and one channel is convolved by only one convolution kernel, thereby greatly reducing the number of parameters of the convolution operation. However, in practical applications, the lack of multiple convolution kernels to jointly extract features reduces the convolution efficiency.

[0087] In order to solve this problem, PConv lightweight convolution was proposed. Figure 4 It can be seen that the Pconv operation is roughly divided into two steps:

[0088] 1) By dividing the input feature map, the number of channels of the input feature map is 32, and only 8 channels are extracted with a 3×3 convolution kernel for ordinary two-dimensional convolution operation. The remaining 24 channels remain unchanged, and the output feature image of 8 channels is output.

[0089] 2) The complete convolution operation is obtained by concatenating the 8-channel feature map output by the convolution operation with the input feature map of the remaining 24 channels according to the channel dimension.

[0090] PConv reduces the number of convolution operations on the channel dimension, thereby ensuring the network execution speed, and splices the output feature map with the input feature map through the channel, thereby avoiding the phenomenon of reducing the convolution efficiency due to the depth-separable convolution. The present invention replaces the second convolution operation used for feature extraction in the two-dimensional convolution of U-Net with PConv, thereby effectively improving the spatial features and ensuring that the UNet network maintains a faster operating speed. By controlling the variables, when only the convolution layer is changed, the training time of the U-Net model is reduced by 4.9% when the Crack500 training set is trained in the same way, which significantly improves the operating efficiency.

[0091] (2.2) Introduce the SCSA spatial-channel attention mechanism.

[0092] Considering that the original U-Net model will lose some information in the process of encoder extracting image features and decoder restoring image features, when facing cracks with different lengths, widths and shapes, the interaction between the feature image space and the channel will be lost, thereby losing important dimensional information. Feature extraction will therefore become less comprehensive and detailed, resulting in blurring at the edges of the cracks. Traditional attention mechanisms such as SENet integrate channel prior information through squeezing, thereby obtaining different receptive fields at the channel level, but SENet only considers channel information and does not pay attention to spatial information. CBAM is the current mainstream space-channel attention mechanism, but CBAM compresses all channel information to enable all spatial information to be shared, but does not consider the synergy between channel and spatial attention.

[0093] To address the synergy between channel and spatial attention, SCSA is proposed by Figure 5 As shown in the schematic diagram of the SCSA space-channel attention mechanism, we can see that SCSA is mainly composed of two modules: shared multi-semantic spatial attention (SMSA) and progressive multi-semantic self-attention (PCSA), with a total of five steps:

[0094] In the first step, the feature map of size B×C×H×W is input, and global average pooling is performed on the height and width dimensions of the feature map. Global average pooling compresses the feature map into a 1×1 feature map by performing average pooling along the channel direction in the two dimensions of the feature map. The number of channels of the feature map is then decomposed through the Spilt operation to obtain two feature maps of different dimensions, namely B×C / 4×H and B×C / 4×W.

[0095] In the second step, the feature maps of two different dimensions are input into the deep convolution with convolution kernels of 3, 5, 7, and 9 respectively, and convolution operations of different sizes are performed. The use of convolution kernels of different sizes can capture different levels of information of the input features.

[0096] In the third step, the two extracted feature maps in different directions are stacked according to the channel dimension to achieve channel splicing, and then normalized within the group. The intra-group normalization is achieved by dividing the feature map into different groups according to the number of channels, and calculating the variance and mean of different intra-group data. The sigmoid activation function can map the input data to between 0 and 1. Two different sets of input feature maps can produce two different feature weights between 0 and 1. These two different feature weights are multiplied with the original input feature map B×C×H×W to complete the construction of the SMSA module.

[0097] In the fourth step, the feature map output by the SMSA module is globally averaged and then normalized for each group. The independence of each sub-feature can be guaranteed by normalizing each group.

[0098] In the fifth step, the normalized feature map is convolved with a 1×1 depth convolution respectively, and the reshape operation of the tensor in pytoch is used to complete the conversion of the basic dimension of the generated feature map, thereby obtaining three vectors V, K, and Q, and V, K, and Q are respectively input into the self-attention mechanism for processing. The process is shown in formula (1):

[0099] (1);

[0100] In the formula, is the output of the self-attention mechanism, is the function of the attention mechanism, is the normalization function, is the transpose of the matrix, is the dimension of the key or query vector, which here refers to the number of features;

[0101] After obtaining the feature image, global average pooling and sigmiod activation function are performed to produce feature weights that are multiplied by the input feature map to output the final feature map and complete the SCSA space-channel attention mechanism.

[0102] It can be understood that the SCSA space-channel attention mechanism can effectively guide channel recalibration. By adding this attention mechanism to the downsampling, upsampling and skip connections of the U-Net model, the feature information between different levels can be effectively combined, making the U-Net model more focused on the cracks, thereby reducing image noise and improving the segmentation ability of the model. When only the SCSA space-channel attention mechanism is added, the U-Net segmentation comparison effect is as follows: Figure 6-Figure 9 ,It can be seen that after adding the SCSA attention mechanism, the segmentation is more refined, the image noise is less, and the network as a whole improves its attention to cracks.

[0103] Specifically, Figure 6 Comparison of the effect of the U-Net network and the network with the SCSA space-channel attention mechanism added to the original image crack-attention experiment. Figure 7 The experimental comparison of the U-Net network and the network with the SCSA spatial-channel attention mechanism added to the original image cracks is shown in Figure 1. Figure 8 The U-Net network and the network with the SCSA space-channel attention mechanism added to the invention are compared in the three-attention experiment on the original image cracks. Fig. 9 The U-Net network and the network with the SCSA space-channel attention mechanism added to the present invention are compared with the experimental results of the four attention levels of the original image cracks. From left to right in the above comparison figure are the original image, the U-Net network attention level graph, and the network attention level graph with the SCSA space-channel attention mechanism added to the present invention;

[0104] (2.3) Reconstruct the downsampling layer.

[0105] After introducing the SCSA spatial-channel attention mechanism, although the overall skeleton segmentation of the crack is obvious, the segmentation of the subtle texture at the edge is still not ideal. This is because U-Net uses the maximum pooling layer as the downsampling layer in the feature compression process.

[0106] The classic U-Net model uses a maximum pooling layer with a receptive field of 2×2 as a downsampling layer. The downsampling layer can aggregate local features and reduce computational overhead. However, in the process of semantic segmentation in the original model, the spatial information of the pooled features in the local area will be lost. Especially in the process of crack segmentation, the feature information area of ​​the crack edge texture is small, and the ordinary pooling operation cannot capture the small area. Therefore, HWD wavelet downsampling came into being. Fig.10 It can be seen from the HWD wavelet downsampling principle shown that HWD wavelet downsampling is divided into two steps as a whole:

[0107] Step 1: Input feature map passes through Haar wavelet transform module HWT to reduce the resolution of feature map while retaining all information of feature map. Haar wavelet transform generates subdivision components in four directions by decomposing high-frequency and low-frequency information of image, thus quadrupling the number of channels, as shown in equations (2) to (5). In the equations, i is the number of rows of the matrix, j is the number of columns of the matrix, and the four subdivision components, LL represents the overall contour and low-frequency information of the image, LH represents horizontal detail information, HL represents vertical detail information, and HH represents diagonal detail information. Through this decomposition method, part of the information can be encoded from the spatial dimension to the channel dimension, thereby obtaining the feature map of wavelet transform;

[0108] Low frequency-low frequency component (LL):

[0109] (2);

[0110] Low-frequency-high-frequency component (LH):

[0111] (3);

[0112] High-frequency-low-frequency component (HL):

[0113] (4);

[0114] High frequency-high frequency component (HH):

[0115] (5);

[0116] In the formula, is the low-frequency-low-frequency component, is the low-frequency-high-frequency component, is the high-frequency-low-frequency component, is the high-frequency-high-frequency component, is the low-frequency residual after removing the high-frequency part of the signal, is the high-frequency residual after removing the low-frequency part of the signal, is the horizontal coordinate of the pixel, is the vertical coordinate of the pixel;

[0117] In the second step, the wavelet transformed feature map is passed into the feature learning module, which consists of convolutional layers, activation functions and batch normalization. The convolutional layer is used to extract features, and the activation function introduces nonlinearity to avoid gradient vanishing in the network. Batch normalization calculates the mean and standard deviation of a batch of data, thereby reducing internal covariate shift and improving the model operation speed, thereby completing the downsampling of the input image size.

[0118] The present invention adopts the HWD wavelet downsampling layer to replace the maximum pooling layer to improve the network's retention of crack edge and fine line features. The present invention designs an ablation experiment to compare the crack segmentation of the U-Net network when only the downsampling layer is replaced. The specific segmentation effect is shown in Figure 11-Figure 14 It can be seen that after replacement downsampling, the network retains more delicate texture features of the crack edge and has less overall image noise, so HWD downsampling is better than maximum pooling. Fig.11 This is the crack edge texture feature preservation image in the original image replacement HWD downsampling experiment comparison. Fig.12 This is the crack edge texture feature preservation image in the HWD downsampling experiment comparison of the original image. Fig.13 This is the crack edge texture feature preservation image in the three-replacement HWD downsampling experiment comparison of the original image. Fig.14 It is a crack edge texture feature retention map in the original image four-replacement HWD downsampling experiment comparison, wherein, from left to right in the above comparison map are the original image, the U-Net network crack edge texture feature retention map, and the crack edge texture feature retention map of the present invention's replacement HWD downsampling;

[0119] (2.4) Reconstruct the upsampling layer.

[0120] Each upsampling operation in the U-Net network is performed in the decoder. Each upsampling layer contains a deconvolution operation, which doubles the size of the feature map. After the deconvolution operation, some convolution, batch normalization, and activation functions are performed to restore the feature information. Recently, due to the introduction of dynamic convolution, upsampling operations based on dynamic convolution have been proposed. Although the effect is better than ordinary upsampling, the dynamic upsampling operation introduces a lot of workload due to time-consuming dynamic convolution and additional subnets used to generate dynamic kernels.

[0121] Therefore, a new upsampler DySample is proposed. DySample is a lightweight and effective dynamic upsampler. Fig.15 The schematic diagram of the DySample lightweight upsampling layer is shown; it can be seen that DySample is divided into two steps as a whole:

[0122] Step 1: As seen Fig.15 As shown in Figure (a), first input the feature with size H×W×C Figure X , through the sampling point generator to generate a sampling component feature map of size sH×sW×2g, which is consistent with the input feature Figure 1 The same is input into the grid_sample function to complete the upsampling. grid_sample is a built-in function of pytorch. It calculates the value of the sampling point by using bilinear interpolation. The sampling method is shown in formula (6):

[0123] (6);

[0124] In the formula, is the output feature map, Sample pixel values ​​from an input image (or batch of images) for a specified coordinate grid in PyTorch. is the input feature map, is the sampling set;

[0125] Step 2: There are two types of sampling point generators in DySample. Fig.15 As shown in Figure (b), the feature map with an input size of H×W×C is passed to the linear projection layer linear. The linear projection layer linear can map the feature map to different dimensions. The working principle of this mapping is to first flatten the feature layer machine, and then map the feature map dimension to different dimensions through matrix multiplication, thereby generating 2gs 2 ×H×W offset feature map, and then perform a Pixel Shuffle operation on the offset feature map to convert the offset into a suitable spatial resolution. PixelShuffle changes the spatial size of the image by rearranging the pixels of each channel into the spatial dimension of the output image and reducing the number of channels. When the pixel shuffle is performed, the input and output channel numbers are C and 2s 2 A linear layer is used to generate a size of 2s 2 ×H×W offset O to generate sH×sW×sC offset network O, the offset network O is the same as the original network The addition is performed to complete the first sampling point generator, see equation (7)-equation (8).

[0126] (7);

[0127] (8);

[0128] In the formula, For the offset network, is a linear layer, is the original network;

[0129] The second method see Fig.15 The difference is that after the linear projection layer, two feature maps of size H×W×2gS2 are output, one of which is reduced to 0.5σ of the original and concatenated with the other feature map of size H×W×2gS2.

[0130] Because by sampling a point for each upsampling position, a point is divided into s2 upsampling points. Therefore, as long as the upsampled s2 points can be dynamically divided, a point can be sampled for each upsampling position, thereby reducing the computational burden and speeding up the sampling efficiency. The semantic segmentation infers the class label of each pixel. Compared with the time taken by the commonly used bilinear difference upsampling, the difference is not much. In the UNet model, in order to obtain high-resolution output, a four-time upsampler is used.

[0131] Accurate per-pixel prediction depends largely on the upsampling quality. For crack segmentation, DySample can improve performance from the internal area and improve boundary quality compared to deconvolution. The DySample upsampler predicts clearer boundaries. In addition, the present invention also designs an ablation experiment to compare the restoration of crack feature details by the U-Net network when only the upsampling layer is replaced. The present invention obtains Figure 16-Figure 18 ,It can be seen that by replacing the U-Net with DySample upsampling, the crack structure is restored more holistically, but the image noise increases. Fig.16 This is the crack structure recovery image in the DySample upsampling experiment comparison after replacing the original image.

[0132] Fig.17 The crack structure recovery image in the DySample upsampling experiment is replaced by the original image 2. Fig.18 This is the crack structure recovery image in the DySample upsampling experiment comparison of the original image.

[0133] Example 3, as another implementation of the present invention, the PHSD-Unet network provided by the embodiment of the present invention is as follows.

[0134] Based on the above improvements, the present invention designs a new semantic segmentation network, named PHSD-Unet network, which improves the robustness of the model for crack segmentation by improving the convolutional blocks, downsampling modules and upsampling modules in the UNet network, introducing a new spatial attention mechanism, and retaining the skip connection and U-shaped structure in the UNet network. Fig.19 , the present invention can see the overall architecture of the network, and the present invention will introduce the overall steps according to the network propagation path below:

[0135] (1) The crack image input first passes through the DoubleConv convolution layer to extract feature information. DoubleConv consists of a two-dimensional convolution and a Pconv, as well as an activation function and a BN layer.

[0136] (2) The feature map output by the DoubleConv convolution layer is input into the SCSA attention mechanism to retain the crack feature information. Part of the output feature layer is input into the HWD for downsampling, and part is retained and input into the decoder of the corresponding layer through the jump connection.

[0137] (3) After the feature map is downsampled by HWD wavelet, the number of channels doubles and the size is reduced to half of the original size. DoubleConv, SCSA, and HWD together constitute each layer of the encoder. The input and output of the feature map are the same as the above steps. After passing through four layers of encoders, the feature extraction of image information is completed.

[0138] (4) The feature map after information extraction has the characteristics of small size and high number of channels. After DySample, the size of the feature map is doubled, and the number of channels remains unchanged.

[0139] (5) The feature map output by DySample passes through a 1×1 convolution layer to complete the compression of the number of channels. The output feature map is twice the size of the original feature map and half the number of channels.

[0140] (6) The feature map output by the 1×1 convolutional layer will be input into the SCSA attention mechanism to extract the backbone information of the feature map. The size and number of channels of the feature map remain unchanged.

[0141] (7) The feature map output by SCSA is input into the DoubleConv convolution layer to complete feature extraction and channel compression.

[0142] (8) The feature map output by the convolution layer of DoubleConv is concatenated with the feature map output by the encoder at the bottom layer. The size and number of channels of the two feature maps are consistent. After concatenation, the number of channels of the feature map is doubled and input into the decoder of the previous layer.

[0143] (9) The decoder of each layer consists of DySample, 1×1Conv, SCSA and DoubleConv. After four times of decoder size and channel number restoration, the image segmentation task is completed.

[0144] Embodiment 4, the embodiment of the present invention provides an improved semantic segmentation network construction system for bridge and road crack identification, the system specifically comprising:

[0145] PConv lightweight convolution reconstruction convolution layer module is used to replace the second convolution operation used for feature extraction in the two-dimensional convolution of U-Net with PConv lightweight convolution, improve spatial features, and improve the UNet network operation speed by controlling variables while only changing the convolution layer;

[0146] The SCSA spatial-channel attention mechanism module is introduced to improve the attention to cracks by adding the channel attention mechanism to the downsampling, upsampling and skip connection of the U-Net model to combine the image feature information between different levels based on the PConv lightweight convolution to blur the edges of cracks.

[0147] The reconstructed downsampling layer module is used to address the small feature information area of ​​the crack edge texture during crack segmentation. Based on the introduction of the SCSA space-channel attention mechanism, the HWD wavelet downsampling layer is used to replace the maximum pooling layer to improve the U-Net network's retention of crack edge and fine line features;

[0148] The reconstructed upsampling layer module is used to reconstruct the upsampling layer based on the introduction of the SCSA space-channel attention mechanism and the use of a new upsampler DySample to complete the construction of an improved semantic segmentation network, complete image segmentation based on the improved semantic segmentation network, and realize crack structure recovery.

[0149] This model experiment uses CRACK500, SDNET2018, and CFD public data sets. The ratio of data set, validation set, and test set is 8:1.5:0.5. The original UNet model, UNet++, Attation-UNet, and the improved PHSD-UNet model of the present invention are compared to verify the accuracy of the new model for crack segmentation. Pixel-level crack detection is essentially a binary classification of each pixel. Precision, recall, F1-score, and intersection over union (IoU) are typical binary classification evaluation indicators. The higher these indicators are, the closer the crack segmentation effect is to the original label. From Table 1, it can be seen that PHSD-Unet has achieved 73.3% in IoU and 82.6% in F1, which are much higher than other models, and from Figure 20-22 ,The present invention can also show that PHSD-Unet is superior to other models in crack segmentation through horizontal comparison.

[0150] Table 1 Comparison of crack segmentation test results

[0151]

[0152] in, Fig. 20 For the original Figure 1 Comparison of crack segmentation diagrams of labels, original UNet model, UNet++, the improved PHSD-UNet model of the present invention, and Attation-UNet. Fig.21 For the original Figure 2Comparison of crack segmentation diagrams of labels, original UNet model, UNet++, the improved PHSD-UNet model of the present invention, and Attation-UNet.

[0153] Fig. 22 For the original Figure 3 The crack segmentation diagram is compared with the label, the original UNet model, UNet++, the improved PHSD-UNet model of the present invention, and Attation-UNet. In the above figure, from left to right are the original image, label, the original UNet model, UNet++, the improved PHSD-UNet model of the present invention, and Attation-UNet.

[0154] The above experiments show the superiority of the new model proposed in the present invention. The introduction of the SCSA attention mechanism module helps the network to better detect cracks of different scales, can effectively eliminate interference information, and improve the ability to extract crack features, thereby accurately segmenting cracks. The reference of the HWD downsampling structure greatly improves the network's segmentation effect on crack edges and textures, reduces image noise, and the DySample upsampling structure can dynamically restore the characteristic morphology of cracks. Compared with the original model, the overall context of the crack structure is clearer, and the PConv partial convolution greatly reduces the calculation time. Compared with different data sets, the model has stronger generalization ability. Therefore, the PHSD-UNet model has higher accuracy in crack detection than the original model, and does not increase much calculation time and memory, so it has higher application value.

[0155] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principles of the present invention should be covered within the protection scope of the present invention.

Claims

1. An improved semantic segmentation network construction method for bridge and road crack identification, characterized in that: This method improves the convolutional blocks, downsampling layers, and upsampling layers in the UNet network, introduces a spatial-channel attention mechanism, retains the skip connections and U-shaped structure in the UNet network, and constructs an improved semantic segmentation network for crack identification. Specifically, the method includes the following steps: S101, PConv lightweight convolution reconstruction convolution layer: replace the second convolution operation used for feature extraction in the two-dimensional convolution of U-Net with PConv lightweight convolution to improve spatial features. By controlling variables, the UNet network operation speed is improved only when the convolution layer is changed; S102, introduces SCSA spatial-channel attention mechanism: Based on PConv lightweight convolution, the channel attention mechanism is added to the downsampling, upsampling and skip connection of the U-Net model to combine the image feature information between different levels and improve the attention to the cracks. S103, reconstruction of downsampling layer: In the crack segmentation process, the feature information area of ​​the crack edge texture is small, so the SCSA space-channel attention mechanism is introduced, and the HWD wavelet downsampling layer is used to replace the maximum pooling layer to improve the U-Net network's retention of crack edge and fine line features; S104, reconstruct upsampling layer: introduce SCSA space-channel attention mechanism, use the new upsampler DySample to reconstruct upsampling layer, build an improved semantic segmentation network, complete image segmentation based on the improved semantic segmentation network, and realize crack structure recovery; In step S104, reconstructing the upsampling layer includes: (a) Input a feature map of size H×W×C, generate a dynamic weight factor S, and output a feature map of sH×sW×sC. The sampling set of the feature map is determined by the weight factor S. Using the sampling point generator, the input features are resampled by the grid sampling function to ensure that the feature map is dynamic, and then spliced ​​with the original feature map to output a feature map of size sH×sW×C, realizing DySample upsampling; (b) The upsampler provides two methods for generating dynamic weight factors S and outputs the weight factors S; In step (b), two methods for generating a dynamic weight factor S include: The first method is to output the input feature map of size H×W×C to the linear connection layer, output a feature map of size H×W×2gS2, and reduce it to 0.25 times of the original size. The reduced O feature map is used as a bias and concatenated with the g feature map to output the weight factor S. The second method is to reduce one of the two feature maps of size H×W×2gS2 to 0.5σ of the original size and concatenate it with the other feature map of size H×W×2gS2, and then perform bias to generate the O feature map, which is concatenated with the g feature map to generate the weight factor S.

2. The improved semantic segmentation network construction method for bridge and road crack identification according to claim 1 is characterized in that: In step S101, PConv lightweight convolution reconstructs the convolution layer, including: (1) The input feature map is divided into 32 channels. 8 channels are extracted and convolved with the convolution kernel. The remaining 24 channels remain unchanged, and the output feature image of 8 channels is output; (2) The 8-channel feature map obtained by the convolution operation is concatenated with the input feature map of the remaining 24 channels to obtain a complete convolution operation.

3. The improved semantic segmentation network construction method for bridge and road crack identification according to claim 1 is characterized in that: In step S102, the SCSA space-channel attention mechanism is introduced, including: In the first step, the input feature map of size B×C×H×W is decomposed along the height and width dimensions using 1D convolution, and global average pooling is applied to each dimension. The feature map is decomposed through the Spilt operation to obtain four sub-features in two different directions. In the second step, the four sub-features in two different directions are input into one-dimensional convolutions with convolution kernels of 3, 5, 7, and 9, respectively, and convolution operations of different scales are performed to extract information of different scales; In the third step, the extracted multi-scale features in two different directions are spliced ​​separately, and then normalized within the group. After the sigmoid activation function, the two different direction features are fused, and the input features and output features are fused in combination with the residual structure to complete the construction of the SMSA module. The fourth step is to perform global average pooling on the feature map output by the SMSA module, and then perform a group normalization operation to ensure the independence of each sub-feature; In the fifth step, the normalized feature map is convolved with 1×1 convolution kernels of different dimensions to obtain V, K, and Q feature images with different numbers of channels. V, K, and Q are respectively input into the self-attention mechanism for feature fusion. After obtaining the feature image, global average pooling and sigmiod function are performed, which are fused with the input feature map, and the final feature map is output to complete the SCSA space-channel attention mechanism.

4. The improved semantic segmentation network construction method for bridge and road crack identification according to claim 1 is characterized in that: In step S103, reconstructing the downsampling layer includes: Step 1: The input feature map passes through the Haar wavelet transform module HWT to reduce the resolution of the feature map while retaining all the information of the feature map. The Haar wavelet transform generates subdivision components in four directions by decomposing the high-frequency and low-frequency information of the image, so that the number of channels becomes four times, and part of the information is encoded from the spatial dimension to the channel dimension to obtain the feature map of the wavelet transform; In the second step, the wavelet transformed feature map is passed into the feature learning module, which consists of a convolutional layer, an activation function, and batch normalization to complete the downsampling of the input image size.

5. The improved semantic segmentation network construction method for bridge and road crack identification according to claim 1 is characterized in that: The method further includes: (1) The crack image is input and passed through the DoubleConv convolution layer to extract feature information. DoubleConv consists of a two-dimensional convolution and a Pconv, activation function and BN layer; (2) The feature map output by the DoubleConv convolution layer is input into the SCSA attention mechanism to retain the crack feature information. Part of the output feature layer is input into the HWD for downsampling, and part is retained and input into the decoder of the corresponding layer through the skip connection; (3) After the feature map is downsampled by HWD wavelet, the number of channels doubles and the size is reduced to half of the original size. DoubleConv, SCSA, and HWD together constitute each layer of the encoder. After the input of the feature map passes through four layers of encoders, the feature extraction of image information is completed; (4) After the information extraction, the feature map has the characteristics of small size and high number of channels. After DySample, the size of the feature map is doubled, and the number of channels remains unchanged; (5) The feature map output by DySample passes through a 1×1 convolution layer to complete the compression of the number of channels. The output feature map is twice the size of the original feature map and half the number of channels. (6) The feature map output by the 1×1 convolutional layer is input into the SCSA attention mechanism to extract the backbone information of the feature map. The size and number of channels of the feature map remain unchanged. (7) The feature map output by SCSA is input into the DoubleConv convolution layer to complete feature extraction and channel number compression; (8) The feature map output by the convolution layer of DoubleConv is concatenated with the feature map output by the encoder at the bottom layer. The size and number of channels of the two feature maps are consistent. After concatenation, the number of channels of the feature map is doubled and input into the decoder of the previous layer. (9) The decoder of each layer consists of DySample, 1×1Conv, SCSA and DoubleConv. After four times of decoder size and channel number restoration, the image segmentation task is completed.

6. An improved semantic segmentation network construction system for bridge and road crack identification, characterized in that: The system implements the improved semantic segmentation network construction method for bridge and road crack identification as claimed in any one of claims 1 to 5, and the system specifically comprises: PConv lightweight convolution reconstruction convolution layer module is used to replace the second convolution operation used for feature extraction in the two-dimensional convolution of U-Net with PConv lightweight convolution, improve spatial features, and improve the UNet network operation speed by controlling variables while only changing the convolution layer; The SCSA spatial-channel attention mechanism module is introduced to improve the attention to cracks by adding the channel attention mechanism to the downsampling, upsampling and skip connection of the U-Net model to combine the image feature information between different levels based on the PConv lightweight convolution to blur the edges of cracks. The reconstructed downsampling layer module is used to address the small feature information area of ​​the crack edge texture during crack segmentation. Based on the introduction of the SCSA space-channel attention mechanism, the HWD wavelet downsampling layer is used to replace the maximum pooling layer to improve the U-Net network's retention of crack edge and fine line features; The reconstructed upsampling layer module is used to reconstruct the upsampling layer based on the introduction of the SCSA space-channel attention mechanism and the use of a new upsampler DySample to complete the construction of an improved semantic segmentation network, complete image segmentation based on the improved semantic segmentation network, and realize crack structure recovery.

Images