Multi-scale three-attention network construction method and device for pixel-level crack segmentation

Abstract

The embodiment of the specification provides a multi-scale three-attention network construction method and device for pixel-level crack segmentation, wherein the method comprises the following steps: adopting a ResNet18 network as a basic network to construct a backbone network; embedding a multi-scale input strategy in the backbone network, detecting feature information through the multi-scale input strategy to obtain a multi-scale feature map; fusing multi-scale feature maps of different resolutions to obtain a fused multi-scale feature map; using feature learning for guiding bottom feature mapping through an AAF block based on the fused multi-scale feature map to obtain an enhanced multi-scale feature map; based on the enhanced multi-scale feature map, detecting attention features through a TA block to obtain an aggregated attention feature map, and guiding training of an end-to-end pixel crack detection network, i.e. a multi-scale three-attention network, based on a crack defect data set according to the aggregated attention feature map through a loss function of the MSTA-Net.

Description

Technical Field

[0001] This document relates to the fields of image processing and artificial intelligence, and in particular to a method and apparatus for constructing a multi-scale three-attention network for pixel-level crack segmentation. Background Technology

[0002] With the continuous development of the modern economy, infrastructure construction has also reached a new level. Among the many types of infrastructure, road construction is the most common, with governments investing significant financial and material resources to comprehensively promote economic development and progress. However, after prolonged exposure to sun and rain and the pressure of passing vehicles, road surfaces are prone to various problems, the most common being cracks. If not repaired by professionals in a timely manner, minor road surface damage will worsen, affecting traffic quality, while more serious damage can lead to road collapse, accidents, and endanger the lives and health of workers. Therefore, in order to extend the service life of roads and prevent accidents, accurate and effective road surface defect detection methods are essential. Traditional crack detection methods have always relied primarily on manual inspection, a time-consuming and labor-intensive process that cannot be applied to modern industrial production processes. Moreover, the inspection process is based on the subjective awareness of the inspectors. When inspectors are mentally fatigued, false or missed cracks are more likely to occur, significantly impacting inspection accuracy. These cracks always face challenging factors such as complex backgrounds, micro-defects, various defect shapes and sizes, and uneven categories, which brings difficulties to accurate crack detection. Therefore, accurate crack detection remains a challenge for some researchers and industrial product manufacturers.

[0003] Currently, after years of research and development, various non-destructive crack detection methods have been adopted. These methods can be broadly categorized into two types: model-based methods and deep learning-based methods. In the early stages of road crack segmentation, model-based segmentation methods were the first to emerge and achieved significant results. Currently, the most popular and widely used model-based methods include threshold-based and edge-detection-based methods, which are the most representative image processing-based detection methods. Henrique et al. proposed a simple unsupervised crack detection system to reduce variance caused by uneven pixels. This method uses morphological filters to preprocess the defect image and a dynamic thresholding method to identify crack pixels. To remove weak edges that do not change significantly with background grayscale, Zhao et al. proposed an improved Canny operator for edge detection. Wavelet transform is used to enhance weak edges in the input image, and the Canny algorithm uses a genetic algorithm to obtain a suitable threshold. Li et al. proposed a crack defect recognition method based on a dual-threshold segmentation approach. The Otsu algorithm is used to remove redundant road markings, and an adaptive threshold iterative segmentation algorithm is proposed to obtain the crack image. Meanwhile, due to the presence of significant noise in the image, the accuracy of segmentation is affected, so morphological denoising methods are used to obtain the target contour. The aforementioned model-based methods always segment cracks from defect images based on specific image features, such as edges, textures, and colors. These features are greatly affected by complex factors such as image noise, shadows, complex backgrounds, and uneven illumination, resulting in poor accuracy and robustness of pixel-level defect detection. In recent years, deep learning-based methods have developed rapidly and have demonstrated excellent performance in various computer vision tasks such as image classification, object detection, and image segmentation. Based on their excellent performance, they have also been widely applied in the detection of road cracks and defects, showing good performance in both detection efficiency and quality. To capture the weak features of road microcracks and eliminate background interference, Liu et al. proposed an end-to-end feature fusion encoder-decoder network containing two new modules to improve crack detection accuracy, and introduced a feature fusion module and a shape semantic prior module into the segmentation network. Qu addressed the problems of detail loss and incomplete crack extraction during road crack detection by proposing a hierarchical feature fusion deep network model based on an attention structure, using a residual network as the base network, to automatically and accurately detect road cracks with uneven intensity distribution. To distinguish the similarity between road surface defects and the background and avoid insufficient segmentation, Gao et al. proposed a pixel-level defect segmentation network that integrates the low, medium, and high levels of the proposed baseline ResNet50 network, more effectively representing detailed defect information. Addressing the issues of insufficient datasets and narrow, elongated shapes in road crack segmentation, Chen proposed a lightweight hybrid convolutional neural network model to achieve road crack segmentation with fewer parameters.While the network model described above has improved the accuracy of pavement crack segmentation to some extent and played a positive role in pavement maintenance and detection, some problems still exist, such as insufficient detection of detailed features and susceptibility to background light. Compared with the previously proposed end-to-end crack detection framework, this study also proposes a network model based on an encoder-decoder structure for image segmentation. The U-Net network is an excellent typical encoder-decoder segmentation network. The encoder and decoder adopt a skip connection structure, which effectively aggregates different feature information from the encoder and decoder, and has been widely used in various segmentation applications such as pavement crack segmentation.

[0004] To address the numerous problems in pavement crack detection, researchers have proposed several variants of the U-Net network structure, further improving its performance. Qu proposed an improved U-Net model to address the complexity of concrete pavement textures and the diversity of crack types. This model employs residual attention in the encoder and a pyramid feature fusion structure in the decoder, effectively segmenting pavement cracks. However, the segmentation network contains a relatively large number of convolutional and pooling layers, which can easily lead to information loss. To solve the problems of pavement crack interruptions and discrete background noise caused by traditional segmentation networks, Han proposed a modified U-Net network to achieve pixel-level semantic segmentation of pavement cracks. However, this network is insensitive to class imbalance in pavement cracks. To overcome the problems of low detection accuracy and poor stability, Qiao proposed an improved U-Net convolutional neural network. This network uses an improved initialization module and an atrous spatial pyramid pooling module, effectively improving accuracy and increasing robustness. However, the network model is not particularly sensitive to crack boundary information, resulting in unclear detected crack boundaries. To focus on pixel-level crack regions, Augustauskas et al. proposed an improved attention-based U-Net network with a spatial pyramid pooling module for pixel-level pavement defect segmentation. In summary, the previously proposed U-Net-based variants have achieved excellent performance in pavement crack segmentation, positively impacting pavement maintenance. However, as the analysis above shows, these networks still have some limitations in high-precision crack segmentation, such as limited receptive field, insufficient local feature processing, and information loss due to pooling operations. Summary of the Invention

[0005] The purpose of this invention is to provide a method and apparatus for constructing a multi-scale three-attention network for pixel-level crack segmentation, aiming to solve the above-mentioned problems in the prior art.

[0006] This invention provides a method for constructing a multi-scale three-attention network for pixel-level crack segmentation, comprising:

[0007] The ResNet18 network is used as the base network to build the backbone network;

[0008] A multi-scale input strategy is embedded in the backbone network, and feature information is detected through the multi-scale input strategy to obtain a multi-scale feature map.

[0009] Multi-scale feature maps of different resolutions are fused to obtain a fused multi-scale feature map.

[0010] Based on the fused multi-scale feature map, feature learning guided by the AAF block is used to obtain an enhanced multi-scale feature map;

[0011] Based on the enhanced multi-scale feature map, an aggregated attention feature map is obtained by detecting attention features through TA blocks, wherein the TA blocks include: spatial attention, channel attention, and pixel attention;

[0012] Using the loss function of MSTA-Net, based on the crack defect dataset, the training of an end-to-end pixel crack detection network, namely a multi-scale three-attention network, is guided by the aggregated attention feature map.

[0013] This invention provides a multi-scale three-attention network construction device for pixel-level crack segmentation, comprising:

[0014] The backbone network construction module is used to build a backbone network based on the crack defect dataset, using a ResNet18 network as the base network.

[0015] A multi-scale input strategy module is used to embed a multi-scale input strategy in the backbone network and obtain a multi-scale feature map by detecting feature information through the multi-scale input strategy.

[0016] The fusion module is used to fuse multi-scale feature maps of different resolutions to obtain fused multi-scale feature maps;

[0017] The enhancement module is used to obtain an enhanced multi-scale feature map by using the AAF block to guide the feature learning of the underlying feature map based on the fused multi-scale feature map;

[0018] The aggregation module is used to obtain an aggregated attention feature map by detecting attention features through TA blocks based on the enhanced multi-scale feature map, wherein the TA blocks include: spatial attention, channel attention, and pixel attention;

[0019] The training module is used to guide the training of the end-to-end pixel crack detection network, i.e., the multi-scale three-attention network, using the loss function of MSTA-Net.

[0020] In this embodiment of the invention, to address the problem of insufficient local feature processing, an AAF block is used to guide feature learning of low-level context, simultaneously acquiring both global and local context. To address the class-imbalanced micro-crack problem, a TA block incorporating spatial attention, channel attention, and pixel attention is used to suppress background and useless information. To support the deep crack segmentation network in acquiring richer feature information, a multi-scale input strategy is also introduced. Furthermore, to enhance multi-scale defect detection capabilities, a multi-scale feature aggregation unit is used to utilize multi-scale context from different receptive fields. This achieves automatic and accurate localization of pixel-level cracks. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in one or more embodiments of this specification or in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of a multi-scale three-attention network construction method for pixel-level crack segmentation according to an embodiment of the present invention;

[0023] Figure 2 This is a schematic diagram of a crack detection scheme according to an embodiment of the present invention;

[0024] Figure 3 This is a schematic diagram of the MST-Net network according to an embodiment of the present invention;

[0025] Figure 4 This is a schematic diagram of the AAF block in an embodiment of the present invention.

[0026] Figure 5 This is a schematic diagram of the TA block in an embodiment of the present invention;

[0027] Figure 6 This is a schematic diagram of a multi-scale three-attention network construction device for pixel-level crack segmentation according to an embodiment of the present invention. Detailed Implementation

[0028] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.

[0029] Method Implementation Examples

[0030] According to embodiments of the present invention, a method for constructing a multi-scale three-attention network for pixel-level crack segmentation is provided. Figure 1 This is a flowchart of a multi-scale three-attention network construction method for pixel-level crack segmentation according to an embodiment of the present invention, such as... Figure 1 As shown, the multi-scale three-attention network construction method for pixel-level crack segmentation according to an embodiment of the present invention specifically includes:

[0031] Step S101: Use the ResNet18 network as the base network to build the backbone network; specifically including:

[0032] Multi-scale feature representation is performed using the 2nd, 3rd, 4th, and 5th residual blocks of the ResNet18 network. The ResNet18 network includes four identical residual blocks, each with a similar network structure. The specific network structure of the residual block consists of two 3×3 convolutional layers followed by a BN layer, a ReLU function, a residual connection, and a max pooling layer.

[0033] Step S102 involves embedding a multi-scale input strategy into the backbone network and obtaining a multi-scale feature map by detecting feature information through the multi-scale input strategy; specifically including:

[0034] The max pooling layer generates a multi-scale input image from top to bottom. The max pooling layer is then processed by a 3×3 convolutional layer, along with a BN layer and a ReLU function, to generate a multi-scale feature map. This multi-scale feature map is then concatenated with the multi-scale feature map from the previous network layer and used as the input to the current network layer.

[0035] Step S103 involves fusing multi-scale feature maps of different resolutions to obtain a fused multi-scale feature map; specifically including:

[0036] Multi-scale feature maps of different resolutions are obtained from the base network, and multi-scale feature maps f1, f2, f3, and f4 with the same feature channels and dimensions are obtained through bilinear interpolation.

[0037] By using feature concatenation, multi-scale feature maps at various scales are fused together. The fused multi-scale feature map contains local feature information and global context information.

[0038] A 1×1 convolutional layer plus a BN layer is used to perform channel adjustment processing on the fused feature map to generate the final fused multi-scale feature map F.

[0039] Step S104: Based on the fused multi-scale feature map, feature learning guided by the AAF block is used to obtain an enhanced multi-scale feature map; specifically including:

[0040] Let F be the fused multi-scale feature map, f i The multi-scale feature maps are obtained after bilinear interpolation. Before feature learning, a 1×1 convolution is performed on the two input multi-scale feature maps to obtain F and f. i For the same channel number, attention weights for the multi-scale feature map are obtained based on Formula 1 using the Mish activation function, 1×1 convolution, and sigmoid activation function:

[0041] F a =σ(Conv 1*1 (Mish(Conv 1*1 (F)+Conv 1*1 (f i )))) Formula 1;

[0042] Among them, F a Represents attention weights, Conv 1*1 This represents a 1×1 convolution, and σ represents the sigmoid activation function.

[0043] Multiply the obtained attention weight by f i This is used for feature learning of the underlying feature maps to enhance feature representation capabilities, and residual connections are introduced in the AAF block according to Equation 2:

[0044] f i ′=Conv 1*1 (f i )*F a +f i Formula 2;

[0045] Among them, f i ′ represents the multi-scale feature map obtained through feature learning.

[0046] Step S105: Based on the enhanced multi-scale feature map, an aggregated attention feature map is obtained by detecting attention features through TA blocks, wherein the TA blocks include: spatial attention, channel attention, and pixel attention; specifically including:

[0047] Step 1: Using the enhanced multi-scale feature map as input to the TA block, channel attention feature map, spatial attention feature map, and pixel attention feature map are obtained; specifically including:

[0048] The channel attention block obtains channel weights through a global average pooling layer, a fully connected layer with a ReLU function, and a fully connected layer with a sigmoid activation function. Based on Equation 3, the sigmoid activation function is used to process the input feature map f. i Multiplying them yields the channel attention feature map:

[0049]

[0050] in, This is the generated channel attention feature map. GAP represents a global average pooling layer, FC represents a fully connected layer that flattens the image information into a vector, and α represents the sigmoid activation function. Indicates element-wise multiplication;

[0051] The global average pooling layer compresses multi-scale feature maps into a single channel, and the spatial attention weights are represented by a 3×3 convolutional layer with a sigmoid activation function, combined with the input feature map f. i The element-wise multiplication of the spatial attention weights is performed, and according to Formula 4, the spatial attention feature map is obtained through the spatial attention block, increasing the weight of the crack region.

[0052]

[0053] in, For the generated spatial attention feature map;

[0054] The pixel attention block, based on Equation 5, passes the spatial weights of each channel through a 1×1 convolutional layer, followed by a sigmoid activation function to rescale the activations to [0,1]. The final pixel attention feature map is then compared with the input feature map f. i 'The result is obtained by element-wise multiplication:

[0055]

[0056] in, This is the generated pixel attention feature map.

[0057] Step 2 involves fusing the aforementioned channel attention feature map, spatial attention feature map, and pixel attention feature map together using element-wise addition to generate an aggregated attention feature map. Specifically, this includes:

[0058] Generate the aggregated attention feature map according to Formula 6:

[0059]

[0060] in, This represents the aggregated attention feature map.

[0061] Step S106 involves training an end-to-end pixel crack detection network, i.e., a multi-scale three-attention network, based on the crack defect dataset and guided by the aggregated attention feature map using the loss function of MSTA-Net. Specifically, this includes:

[0062] Based on Equation 7, the training of the end-to-end pixel crack detection network, i.e., the multi-scale three-attention network, is guided by the loss function of MSTA-Net:

[0063]

[0064] Where p and t represent the predicted value and the actual value, respectively, and Loss i (,t) represents the loss function of MSTA-Net, where i represents the number of pixels, H represents the image height, W represents the image width, and t i p represents the true value of the i-th pixel. i This represents the predicted value of the i-th pixel;

[0065] A deep supervision mechanism is applied to each output layer. 1×1 convolutions are performed on the four branches using the softmax function to generate four output defect masks. The BCE loss function is applied to the four network branches and the final output network branch. The expression for the total loss function is determined according to Equation 8.

[0066]

[0067] Where Loss1 represents the first network branch loss, Loss2 represents the second network branch loss, Loss3 represents the third network branch loss, Loss4 represents the fourth network branch loss, and Loss5 represents the final output network branch loss.

[0068] The technical solutions of the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0069] This invention proposes a multi-scale three-attention network, called MST-Net, for end-to-end and precise pixel crack detection, serving quality control and maintenance decision-making. Figure 2 This is a schematic diagram of a crack detection scheme according to an embodiment of the present invention. Combining three publicly available crack defect datasets, this method demonstrates excellent performance in pixel-level crack detection under complex background interference, and is particularly suitable for the accurate detection of microcracks.

[0070] MST-Net of the present invention embodiment, such as Figure 3 As shown, it starts with a 7×7 convolutional layer, followed by a batch normalization (BN) layer and a rectified linear unit (ReLU) to obtain a large receptive field.

[0071] Residual Network: The residual network is a typical deep network that has been widely used in various models as a base network. It exhibits excellent feature representation performance in different tasks such as image classification, object detection, and image segmentation. Due to the success of the residual network in computer vision, the ResNet18 network is used as the base network to construct the backbone network. Specifically, the 2nd, 3rd, 4th, and 5th residual blocks of the ResNet18 network are used for multi-scale feature representation.

[0072] like Figure 3 As shown, the residual network consists of four identical residual blocks, each with a similar network structure, separated by two 3×3 convolutional layers, followed by a BN layer, a ReLU function, residual connections, and a max pooling layer.

[0073] Multi-scale input units: To support the deep slot segmentation network in acquiring richer feature information, a multi-scale input strategy is embedded in the backbone network to detect abundant feature information. Simultaneously, it can significantly reduce the impact of information loss caused by pooling operations. Here, the multi-scale input image from top to bottom is obtained through average pooling layers. The pooling layers are then processed by 3×3 convolutional layers, accompanied by BN layers and ReLU functions, generating multi-scale feature maps through the convolutional layers. These feature maps are concatenated with feature maps from the previous network layer, serving as the input to the current network layer.

[0074] Multi-scale Feature Learning: To ensure the detection accuracy of multi-scale defects that integrate global and local feature information, effective multi-scale feature aggregation demonstrates significant performance optimization. Inspired by this mechanism, feature maps of different resolutions are obtained from the base network. Through bilinear interpolation, feature maps f1, f2, f3, and f4 with the same feature channels and dimensions are obtained. Based on this, feature concatenation is used to fuse the feature maps at different scales, incorporating both local feature information and global contextual information. The fused feature map is then used by the AAF block to guide the feature learning of the underlying feature maps.

[0075] To reduce computational costs, a 1×1 convolutional layer + BN layer is used to perform channel adjustment processing on the fused feature map to generate the final fused feature F. This provides additional attention fusion guidance for each scale of the feature map, making the extracted features more discriminative.

[0076] To address the coarse feature maps generated by residual networks and to make the extracted features easier to identify, a feature map additive attention fusion guided method based on AAF blocks is proposed.

[0077] like Figure 4 As shown, F is a multi-scale fused feature map, which contains not only shallow texture and boundary information, but also high-level semantic information. iThis is the feature map after bilinear interpolation. Before feature learning, a 1×1 convolution is performed on the two input feature maps to obtain F and fi with the same channel number. These two feature maps are processed by element-wise addition. Attention weights for the feature maps are obtained through the Mish activation function, 1×1 convolution, and sigmoid activation function.

[0078] F a =σ(Conv 1*1 (Mish(Conv 1*1 (F)+Conv 1*1 (f i )))) (1)

[0079] Where F a Represents attention weights, Conv 1*1 σ represents a 1×1 convolution, and σ represents the sigmoid activation function.

[0080] He multiplied the attention weight he received by f i This is used for feature learning of the underlying feature map to enhance feature representation capabilities. Meanwhile, residual connections are also introduced into the proposed AAF block, as shown in equation (2).

[0081] f i ′=Conv 1*1 (f i )*F a +f i (2)

[0082] Where f i ' represents the feature map obtained through feature learning. Combining the proposed AAF block yields an enhanced multi-scale feature map, which can be used for feature decoding to obtain more accurate segmentation results.

[0083] The multi-scale feature learning unit (MFL) of this invention generates output feature maps containing shallow and deep contextual information at different scales. However, simple element-wise addition or channel feature concatenation cannot effectively aggregate multi-scale feature maps because they not only generate too many parameters but also lead to feature information redundancy. Therefore, to achieve effective feature representation of crack images and suppress background interference, a attention block (TA) is proposed to detect attention features, thereby better focusing on the crack region. The proposed TA block is as follows: Figure 5 As shown, the TA block consists of spatial attention, channel attention, and pixel attention.

[0084] like Figure 5As shown, the output feature map of the MFL unit serves as the input to the TA block. The TA block contains three parallel network branches, focusing on key channel information, spatial information, and pixel information. These attention feature maps are then fused together using element-wise addition to generate the final attention feature map.

[0085] Channel Attention: To better focus on key feature channels, this study proposes a channel attention block to construct the interdependencies between channels. Here, channel weights are obtained by a global average pooling layer, a fully connected layer with a ReLU function, and a fully connected layer with a sigmoid activation function. The sigmoid activation function is used to apply the input feature map f... i The channel attention feature map is obtained by multiplying the two sides, as shown in equation (3).

[0086]

[0087] in This is the generated channel attention feature map. GAP represents a global average pooling layer, FC represents a fully connected layer that flattens the image information into a vector, and α represents the sigmoid activation function. This represents element-wise multiplication.

[0088] By using channel attention blocks, different spatial locations in the feature channels have the same weight, the interdependencies between channels are obtained, and useless feature channels are suppressed.

[0089] Spatial attention: In order to express the information of the crack region more effectively, a spatial attention block is proposed to obtain the spatial attention feature map and increase the weight of the crack region, as shown in Equation (4).

[0090]

[0091] in For the generated spatial attention feature map.

[0092] Global average pooling layers can compress feature maps into a single channel, and spatial attention weights can be effectively represented by 3×3 convolutions with sigmoid activation. Combined with the input feature map f i The element-wise multiplication of the spatial attention weights generates a spatial attention feature map, highlighting the defective regions.

[0093] Pixel Attention: For a pixel attention block, the spatial weights of each channel can be obtained by using a 1×1 convolutional layer, followed by a sigmoid activation function to rescale the activation to [0,1], as shown in Equation (5). The final pixel attention feature map is obtained by comparing it with the input feature map f. i It is obtained by element-wise multiplication.

[0094]

[0095] in This is the generated pixel attention feature map. Pixel attention blocks emphasize the importance of each pixel in the feature map, further highlighting detailed feature information.

[0096] Since the attention points of each attention mechanism are different in the task of road surface crack segmentation, the above three attention blocks are merged together to form the TA block proposed in this study, and the specific expression is shown in Equation (6).

[0097]

[0098] in This represents the aggregated attention feature map. Combined with the proposed TA block, it effectively integrates spatial attention, channel attention, and pixel attention information, suppressing background interference and providing effective support for accurate segmentation of road surface cracks with class imbalance problems.

[0099] Deep supervision: The binary cross-entropy (BCE) loss function has shown good performance in various classification tasks, representing the difference between the true and predicted values. Based on the excellent performance of the BCE loss function, it is set as the loss function of the proposed MSTA-Net to guide network training, as shown in Equation (7).

[0100]

[0101] Where p and t represent the predicted value and the actual value, respectively.

[0102] To improve the segmentation accuracy of the MST-Net network and accelerate the convergence speed of the segmentation network, a deep supervision mechanism was also adopted for each output layer.

[0103] For the deep monitoring mechanism, 1×1 convolutions are performed using the softmax function on the four branches to generate four output defect masks. The BCE loss function is applied to the four network branches and the final output network branch. The special expression for the total loss function is shown in Equation (8).

[0104] Loss=Loss1+Loss2+Loss3+Loss4+Loss5 (8)

[0105] The deep supervision mechanism proposed in this study can not only alleviate the problem of excessive parameters, but also effectively accelerate the convergence speed of the crack segmentation model.

[0106] Currently, intelligent crack detection is of significant value for infrastructure maintenance, especially roads in China. Accurate crack detection enables timely repair and maintenance of road surfaces, greatly reducing the occurrence of hazards. However, due to the complex background, micro-defects, varying shapes and sizes, and imbalanced categories of road defects, crack detection remains a major challenge. In recent years, deep learning has demonstrated excellent performance in pixel-level image segmentation, but it still faces challenges in demanding pixel-level segmentation, such as limited receptive field, insufficient local feature processing, and information loss due to pooling operations. To address these issues, a multi-scale three-attention network, MST-Net, is proposed for end-to-end pixel-level crack detection. First, a multi-scale input strategy is applied to the proposed segmentation network to capture more contextual information. Simultaneously, it effectively reduces the impact of information loss caused by pooling operations. Second, to achieve effective representation of local features, an additive attention fusion (AAF) block is proposed to guide feature learning and capture both global and local context. Furthermore, for crack detection tasks with imbalance problems, a three-attention (TA) block is proposed to detect spatial, channel, and pixel attention information respectively, suppressing background and useless information, which is beneficial for the characterization of microcracks. Finally, to address the problem of limited defect receptive domain, a multi-scale feature aggregation unit is proposed for feature fusion to improve the detection capability of multi-scale defects. To better guide network training, a deep supervision mechanism is also introduced to accelerate the convergence speed of the proposed segmentation model and improve the performance of defect segmentation.

[0107] Device Examples

[0108] According to embodiments of the present invention, a multi-scale three-attention network construction device for pixel-level crack segmentation is provided. Figure 6 This is a schematic diagram of a multi-scale three-attention network construction device for pixel-level crack segmentation according to an embodiment of the present invention, as shown below. Figure 6 As shown, the multi-scale three-attention network construction device for pixel-level crack segmentation according to an embodiment of the present invention specifically includes:

[0109] Backbone network construction module 60 is used to construct a backbone network based on a crack defect dataset, using a ResNet18 network as the base network.

[0110] The multi-scale input strategy module 61 is used to embed a multi-scale input strategy in the backbone network and obtain a multi-scale feature map by detecting feature information through the multi-scale input strategy.

[0111] The fusion module 62 is used to fuse multi-scale feature maps of different resolutions to obtain fused multi-scale feature maps;

[0112] Enhancement module 63 is used to obtain an enhanced multi-scale feature map by using the AAF block to guide the feature learning of the underlying feature map based on the fused multi-scale feature map;

[0113] The aggregation module 64 is used to obtain an aggregated attention feature map by detecting attention features through TA blocks based on the enhanced multi-scale feature map, wherein the TA blocks include: spatial attention, channel attention and pixel attention;

[0114] Training module 65 is used to guide the training of the end-to-end pixel crack detection network, i.e., the multi-scale three-attention network, using the loss function of MSTA-Net.

[0115] The embodiments of the present invention are device embodiments corresponding to the above method embodiments. The specific operation of each module can be understood with reference to the description of the method embodiments, and will not be repeated here.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing a multi-scale three-attention network for pixel-level crack segmentation, characterized in that, include: The ResNet18 network is used as the base network to build the backbone network; A multi-scale input strategy is embedded in the backbone network, and feature information is detected through the multi-scale input strategy to obtain a multi-scale feature map. Multi-scale feature maps of different resolutions are fused to obtain a fused multi-scale feature map. Specifically, this involves: obtaining multi-scale feature maps of different resolutions from the base network; performing bilinear interpolation to obtain multi-scale feature maps f1, f2, f3, and f4 with the same feature channels and dimensions; fusing the multi-scale feature maps at different scales through feature concatenation, resulting in a fused multi-scale feature map containing local feature information and global context information; and applying a 1 × 1 convolutional layer + BN layer to adjust the channels of the fused feature map to generate the final fused multi-scale feature map F. Based on the fused multi-scale feature map, feature learning guided by the AAF block is used to obtain an enhanced multi-scale feature map; specifically including: Let F be the fused multi-scale feature map, f i The multi-scale feature maps are obtained after bilinear interpolation. Before feature learning, a 1 × 1 convolution is performed on the two input multi-scale feature maps to obtain F and f. i For the same channel number, attention weights for the multi-scale feature map are obtained based on Formula 1 using the Mish activation function, 1 × 1 convolution, and sigmoid activation function: Official 1; in, Indicates attention weights, σ represents a 1 × 1 convolution, and σ represents the sigmoid activation function; Multiply the obtained attention weight by f i This is used for feature learning of the underlying feature maps to enhance feature representation capabilities, and residual connections are introduced in the AAF block according to Equation 2: Official 2; in, This represents a multi-scale feature map obtained through feature learning; Based on the enhanced multi-scale feature map, an aggregated attention feature map is obtained by detecting attention features through TA blocks, wherein the TA blocks include: spatial attention, channel attention, and pixel attention; Using the loss function of MSTA-Net, based on the crack defect dataset, the training of an end-to-end pixel crack detection network, namely a multi-scale three-attention network, is guided by the aggregated attention feature map.

2. The method according to claim 1, characterized in that, Using ResNet18 as the base network to build the backbone network specifically includes: Multi-scale feature representation is performed using the 2nd, 3rd, 4th, and 5th residual blocks of the ResNet18 network. The ResNet18 network includes four identical residual blocks, each with a similar network structure. The specific network structure of the residual block consists of two 3 × 3 convolutional layers followed by a BN layer, a ReLU function, a residual connection, and a max pooling layer.

3. The method according to claim 2, characterized in that, Embedding a multi-scale input strategy in the backbone network, and obtaining a multi-scale feature map by detecting feature information through the multi-scale input strategy, specifically includes: The max pooling layer generates a multi-scale input image from top to bottom. The max pooling layer is then processed by a 3 × 3 convolutional layer, along with a BN layer and a ReLU function, to generate a multi-scale feature map. This multi-scale feature map is then concatenated with the multi-scale feature map from the previous network layer and used as the input to the current network layer.

4. The method according to claim 1, characterized in that, Based on the enhanced multi-scale feature map, the aggregated attention feature map is obtained by detecting attention features through TA blocks, specifically including: Using the enhanced multi-scale feature map as input to the TA block, channel attention feature map, spatial attention feature map, and pixel attention feature map are obtained; The channel attention feature map, spatial attention feature map, and pixel attention feature map are fused together by element-wise addition to generate an aggregated attention feature map.

5. The method according to claim 4, characterized in that, Using the enhanced multi-scale feature map as input to the TA block, the resulting channel attention feature map, spatial attention feature map, and pixel attention feature map specifically include: The channel attention block obtains channel weights through a global average pooling layer, a fully connected layer with a ReLU function, and a fully connected layer with a sigmoid activation function. Based on Equation 3, the sigmoid activation function is used to process the input feature map. Multiplying them yields the channel attention feature map: Official 3; in, It is the generated channel attention feature map. This indicates a global average pooling layer. This represents a fully connected layer that flattens image information into vectors. This represents sending the sigmoid activation function. Indicates element-wise multiplication; The global average pooling layer compresses multi-scale feature maps into a single channel, and spatial attention weights are represented by a 3×3 convolutional layer with a sigmoid activation function, combined with the input feature map. The element-wise multiplication with the spatial attention weights, according to Formula 4, involves obtaining the spatial attention feature map through the spatial attention block and increasing the weight of the crack region. Official 4; in, For the generated spatial attention feature map; The pixel attention block, based on Equation 5, passes the spatial weights of each channel through a 1 × 1 convolutional layer, followed by a sigmoid activation function to rescale the activations to [0,1]. The final pixel attention feature map is then compared with the input feature map. Element-wise multiplication yields: Official 5; in, This is the generated pixel attention feature map.

6. The method according to claim 5, characterized in that, The aforementioned channel attention feature map, spatial attention feature map, and pixel attention feature map are fused together using element-wise addition to generate an aggregated attention feature map. This process specifically includes: Generate the aggregated attention feature map according to Formula 6: Official 6; in, This represents the aggregated attention feature map.

7. The method according to claim 1, characterized in that, The training of the end-to-end pixel crack detection network, i.e., the multi-scale three-attention network, guided by the loss function of MSTA-Net, specifically includes: Based on Equation 7, the training of the end-to-end pixel crack detection network, i.e., the multi-scale three-attention network, is guided by the loss function of MSTA-Net: Official 7; in, and These represent the predicted value and the actual value, respectively. This represents the loss function of MSTA-Net, where i represents the number of pixels, H represents the image height, W represents the image width, and t represents the image height. i p represents the true value of the i-th pixel. i This represents the predicted value of the i-th pixel; A deep supervision mechanism is applied to each output layer. 1 × 1 convolutions are performed using the softmax function on the four branches to generate four output defect masks. The BCE loss function is applied to the four network branches and the final output network branch. The expression for the total loss function is determined according to Equation 8. Formula 8; Where Loss1 represents the first network branch loss, Loss2 represents the second network branch loss, Loss3 represents the third network branch loss, Loss4 represents the fourth network branch loss, and Loss5 represents the final output network branch loss.

8. A multi-scale three-attention network construction device for pixel-level crack segmentation, characterized in that, include: The backbone network construction module is used to build a backbone network based on the crack defect dataset, using a ResNet18 network as the base network. A multi-scale input strategy module is used to embed a multi-scale input strategy in the backbone network and obtain a multi-scale feature map by detecting feature information through the multi-scale input strategy. The fusion module is used to fuse multi-scale feature maps of different resolutions to obtain fused multi-scale feature maps. Specifically, it is used to: obtain multi-scale feature maps of different resolutions from the base network, and through bilinear interpolation, obtain multi-scale feature maps f1, f2, f3, and f4 with the same feature channels and dimensions; and fuse the multi-scale feature maps at each scale through feature concatenation. The fused multi-scale feature map contains local feature information and global context information. A 1 × 1 convolutional layer + BN layer is used to perform channel adjustment processing on the fused feature map to generate the final fused multi-scale feature map F; The enhancement module is used to obtain enhanced multi-scale feature maps by using the AAF block to guide feature learning from the fused multi-scale feature maps; specifically, it is used for: Let F be the fused multi-scale feature map, f i The multi-scale feature maps are obtained after bilinear interpolation. Before feature learning, a 1 × 1 convolution is performed on the two input multi-scale feature maps to obtain F and f. i For the same channel number, attention weights for the multi-scale feature map are obtained based on Formula 1 using the Mish activation function, 1 × 1 convolution, and sigmoid activation function: Official 1; in, Indicates attention weights, σ represents a 1 × 1 convolution, and σ represents the sigmoid activation function; Multiply the obtained attention weight by f i This is used for feature learning of the underlying feature maps to enhance feature representation capabilities, and residual connections are introduced in the AAF block according to Equation 2: Official 2; in, This represents a multi-scale feature map obtained through feature learning; The aggregation module is used to obtain an aggregated attention feature map by detecting attention features through TA blocks based on the enhanced multi-scale feature map, wherein the TA blocks include: spatial attention, channel attention, and pixel attention; The training module is used to guide the training of the end-to-end pixel crack detection network, i.e., the multi-scale three-attention network, using the loss function of MSTA-Net.

Images