Road crack detection method and system based on cross-attention guided feature alignment network

Abstract

The present invention discloses a road crack detection method and system based on a cross-attention guided feature alignment network. First, a data set is acquired and then input into an automated road crack detection model, which outputs crack detection results. The automated road crack detection model includes an encoding stage and a decoding stage. The encoding stage uses a weight-sharing collaborative network to downsample the input crack image, extracting rich multi-scale feature information. A cross-layer interaction module is also used to promote feature transfer and feature richness, thereby improving the network's ability to utilize features from different levels and represent semantic information. The decoding stage uses a multi-scale feature fusion and feature alignment module to fuse and align features from different levels or scales, enhancing the segmentation network's ability to perceive details and boundaries and improving the precision and accuracy of the segmentation results. This method can efficiently extract and fuse multiple features, thereby improving the accuracy of road crack detection.

Description

Technical Field

[0001] The present invention belongs to the field of road crack detection, and more particularly, relates to a method for detecting road cracks. Background Art

[0002] Roads play a vital role in economic prosperity and expansion, and they provide significant social benefits. Road networks facilitate transportation and connectivity, providing convenient access to employment, social, health, and education services. Therefore, road infrastructure is considered one of the most important of all public assets. However, due to various factors such as location, traffic volume, weather, and construction materials, road surfaces can gradually wear and deteriorate over time, leading to traffic accidents. In many countries, road maintenance mileage accounts for 99% of the total road mileage. Therefore, road crack detection is a key task in the road infrastructure sector, involving the location and classification of road cracks. It can identify roads requiring maintenance to reduce potential safety hazards and is particularly important for effective and economical road maintenance and traffic safety.

[0003] In traditional methods, road crack areas are usually detected using thresholding methods. These algorithms can quickly detect the results of the input image by setting different thresholds. Because road crack areas absorb more light, they usually appear as darker areas in the image. However, when there is a certain degree of noise, pixels with intensity lower than the damaged pixel seriously reduce the overall detection performance. These methods lack the description of global information, are sensitive to noise, and mainly rely on the selection of thresholds. Other researchers use manually designed feature descriptors to identify damage in images. For example, Gabor filters and wavelet transforms have made significant progress in detecting simple cracks. However, due to the complex topology, arbitrary shapes and widths, oil stains, weeds, stains and other strong interferences on the road, the performance of these methods is still limited.

[0004] With the further development of machine learning methods, they have been widely applied in the field of road crack detection. Improved active contour models and support vector machines (SVMs) based on greedy search have been used in bridge damage detection research. Some researchers have proposed an SVM-based method to compute a probability map with multi-scale neighborhood information. Through a fusion algorithm, multiple probability maps obtained from probabilistic generative models (PGMs) and SVM methods are combined into a fused map, which can identify cracks more accurately than any individual probability map. Other researchers have used random forest methods to classify multiple spatially adjusted visual features. However, these detection methods can only detect cracks that have already been learned and are therefore unable to detect new cracks. To address this issue, CrackForest, based on a random structured forest, proposed an automatic crack detection method that effectively suppresses noise by selecting crack features and learning internal structures. However, it does not consider different damage categories in complex crack extraction scenarios. Because traditional methods simulate cracks by manually setting color or texture features, they can only meet the requirements of crack extraction in specific situations. These methods cannot perform robust detection in changing environments. Consequently, manually designed features are inefficient in complex situations and cannot be used to extract cracks from diverse road images.

[0005] Recent theoretical developments have demonstrated that deep learning can solve complex problems by automatically learning features from different layers. Deep convolutional neural networks (DCNNs), with their rich hierarchical features and end-to-end trainable framework, have made significant progress in pixel-level semantic segmentation. Recently, several crack detection methods have been proposed based on object detection, image patch segmentation, and deep learning. However, these crude estimation methods are unable to extract cracks at the pixel level, making it difficult to accurately classify cracks and assign severity levels. Some researchers have proposed a solution for pixel-level crack extraction using FCN networks. However, this approach fails to consider that cracks of varying widths and topologies require different amounts of contextual information. Furthermore, this approach ignores the diverse contributions of crack features to crack detection and treats all crack features in the same manner. Some researchers have implemented the DeepCrack network based on the encoder-decoder architecture of SegNet, merging the convolutional features generated by the encoder and decoder networks at the same scale to achieve pixel-level crack detection. However, based on the SegNet architecture, the features learned in the encoder-decoder stage are relatively simple, and much of the spatial information is lost during upsampling, making it impossible to recover in shallow layers. Researchers have also developed a crack segmentation network within the DeepLabv3 framework for pixel-accurate tunnel crack segmentation. While this method leverages the Atrous Spatial Pyramid Pooling (ASPP) module to capture multi-scale information, it fails to fully recognize the importance of upsampling for improving detection results. Deep learning methods achieve better results than traditional methods. However, research on trainable DCNN models for robust pixel-level crack detection with rich semantic information remains insufficient. These DCNN-based crack detection methods do not consider crack classification and damage severity. The use of deep learning features in these detection methods significantly improves the detection performance of road crack detection. However, they still face several important challenges. First, many types of road cracks exist, such as cracks, potholes, shoulder damage, and roadside facilities. Each damage type has unique characteristics and manifestations, making it difficult to learn effective multi-scale features for efficient road crack detection. Second, road cracks occur in varying sizes and shapes, highlighting the importance of utilizing multiple feature maps with varying resolutions to capture features at different scales. Despite the progress, existing methods still face challenges in comprehensively understanding the importance of different regions in road images and effectively utilizing relevant features to improve the detection performance of road crack detection tasks. Summary of the Invention

[0006] In response to the shortcomings of existing technologies and the need for improvement, this paper provides a road crack detection method based on a cross-attention guided feature alignment network, which can accurately and automatically detect road cracks. The method includes the following steps:

[0007] Step 1: Use road crack images collected by a vehicle-mounted high-definition camera as a dataset and divide the actual road crack images into a training set and a test set;

[0008] Step 2: Build an automated crack detection model consisting of an encoder and a decoder. The encoder includes a weight-sharing collaborative network module (encoder1 and encoder2) and a cross-layer interaction module. The decoder includes a multi-scale feature fusion and feature alignment module.

[0009] The weight-sharing collaborative network includes a multi-scale image block embedding layer and a multi-head convolutional self-attention layer, and encoder 1 and encoder 2 each have K stages. The crack image is input and feature maps of different scales of K stages are obtained respectively;

[0010] The cross-layer interaction module first reshapes and swaps the dimensions of feature maps of different scales in the same stage, then normalizes and globally averages the resulting tensors, and finally uses a self-attention mechanism to extract more comprehensive semantic information about crack features, improving the network's multi-scale feature expression capabilities.

[0011] The multi-scale feature fusion module adopts a dual progressive fusion strategy of reconstruction and upsampling blocks to fuse the extracted features containing rich multi-scale semantic information;

[0012] The input of the feature alignment module is the output image of the multi-scale fusion module. The purpose is to combine the context information in the image to ensure the scale and semantic consistency between features at different levels or branches, thereby improving the detection ability of small cracks.

[0013] Step 3: Use the training set in step 1 to train the automatic crack detection model in step 2, and use the trained model to realize road crack detection.

[0014] Furthermore, step 1 includes:

[0015] 11) Using road crack images collected by a high-definition camera on a vehicle as a dataset, a binary label image is created based on the crack images in the dataset, where each pixel in the label image represents a category, either crack or background;

[0016] 12) The road crack images and their corresponding binary label images are combined into a road crack automatic detection image dataset, and the dataset is divided into a training set and a test set according to a certain ratio.

[0017] Furthermore, the specific implementation of the weight-sharing collaborative network processing process in step 2 includes the following sub-steps:

[0018] 211) Given an input original image F1, first use a backbone block consisting of two convolutional layers to adjust the image size and number of channels to obtain the input features of the multi-scale image block embedding layer in the first stage;

[0019] 212) Input features are fed into encoder1 and encoder2 simultaneously for feature extraction. Each stage includes a multi-scale image block embedding layer and a multi-head convolutional self-attention layer to extract crack information.

[0020] 213) The multi-scale image patch embedding layer utilizes fine-grained and coarse-grained visual markers at the same feature level using convolution operations with overlapping patches; specifically, for the 2D reshaped output feature map from the previous stage As the input of stage i, we first learn a function F k×k (·), X i Mapped to have channel size C i New mark F k×k (X i ), where F(·) is a 2D convolution operation with a kernel size of k×k, a stride of s, and a padding of p; the output 2D label map With the following height and width: The multi-scale image patch embedding layer adjusts the length of the labeled sequence by changing the stride and padding, outputting features of the same resolution size with different label sizes;

[0021] 214) The labeled feature maps of different scales obtained by the multi-scale image block embedding layer are respectively input into each self-attention layer of the multi-head convolutional self-attention layer to obtain global feature maps of different scales; the features of the last scale among the features output by the multi-scale image block embedding layer are input into the convolutional local feature block to obtain a local feature map; finally, the local features and global features are aggregated using the multi-scale feature interaction block, and the output feature map of the first stage is obtained through a convolution operation;

[0022] Each stage of 215) is processed according to steps 213)-215), and encoder1 and encoder2 respectively output multiple feature maps of different scales.

[0023] Furthermore, the number of channels of the first convolutional layer in the backbone block is C1 / 2, and the number of channels of the second convolutional layer is C1. The size of the feature map generated by the two convolution operations is H / 4×W / 4×C1, where C1 is the channel size of the first stage. Each convolution is followed by Batch Normalization and Hardswish activation function.

[0024] Furthermore, the processing process of the multi-head convolutional self-attention layer is as follows: the input is first normalized by Layer Norm, then self-attention calculation is performed through Factorized MHSA, and then the original input and attention output are added through residual connection, and then Layer Norm normalization is performed. After that, nonlinear transformation is performed through FFN, and then residual connection is performed again, and finally Layer Norm normalization is performed again; the convolutional local feature block is implemented using a residual bottleneck block consisting of 1×1 convolution, 3×3 depth convolution and 1×1 convolution.

[0025] Furthermore, the specific implementation of the cross-layer interaction module in step 2 includes the following sub-steps:

[0026] 221) reshape the input feature matrices whose feature shapes are [B, C, H, W] and [B, 2C, H, W], respectively, converting them from 4D tensors to 3D tensors and swapping their second and third dimensions to obtain tensors e and r, respectively; [B, C, H, W] represents the collective name of the feature maps obtained at each stage in encoder1, and [B, 2C, H, W] represents the collective name of the feature maps obtained at each stage in encoder2. The number of channels between the feature maps of encoder1 and encoder at the same stage is twice that of the other.

[0027] 222) Perform normalization and global average pooling operations on tensors e and r simultaneously to obtain feature vectors of length 1, denoted as e_t and r_t;

[0028] 223) Map e_t and r_t to each other's dimensions, and insert a dimension of length 1 on the second dimension;

[0029] 224) Concatenate r and e_t, e and r_t in the second dimension respectively, and then pass the concatenated feature tensors into the Transformer encoder at the same time for their respective cross-attention calculations, and finally obtain multiple feature maps, denoted as P1, P2, ..., P k .

[0030] Furthermore, in step S2, the cross-attention calculation formula of the cross-layer interaction module is:

[0031]

[0032] in, represents the query vector matrix, where m represents the number of query vectors, d k represents the dimension of the key vector; represents the key vector matrix, where n represents the number of key vectors, d k represents the dimension of the key vector; represents a value vector matrix, where d v Represents the dimension of the key vector; QK T Represents the dot product matrix between the query vector and the key vector, whose dimension is m×n; softmax() represents the softmax function, which is used to normalize the score to a probability value; Used to normalize the dot product to prevent the dot product value from being too large or too small.

[0033] Furthermore, the cross-layer interaction module obtains K feature maps of different scales. When K = 4, the multi-scale feature fusion module implements the fusion processing of K feature maps through the following steps:

[0034] In the first step, the four feature maps P1, P2, P3, and P4 obtained by the cross-layer interaction module are concatenated in their respective first dimensions, and then four new feature maps are generated through convolution operations, which are recorded as A, B, C, and D.

[0035] In the second step, D is upsampled and then convolved to obtain a new feature map D', and then C and D' are concatenated in the first dimension and then convolved to obtain a feature map D";

[0036] In the third step, D' is upsampled and then convolved to obtain a new feature map D'', and then B and D'' are concatenated in the first dimension and then convolved to obtain the feature map B';

[0037] The fourth step is to upsample B' and then convolve it to get the new feature map B", then concatenate A and B' in the first dimension and then convolve them to get the feature map A';

[0038] In the fifth step, a convolutional layer is used to reduce the number of channels of the feature maps D'', B', and A', and the reduced feature maps are upsampled to obtain three new feature maps P5, P6, and P7;

[0039] Therefore, after the above five-step multi-scale fusion operation, four new feature maps are obtained: P1, P5, P6, and P7. These four new feature maps are the input of the feature alignment module.

[0040] Furthermore, the feature alignment module in step S2 performs feature alignment on the four outputs of the multi-scale feature fusion module. The process is to align P1 with P5, P6, and P7 respectively to obtain three new feature maps, and then merge P1 and these three new feature maps to obtain the final output feature map of the feature alignment module. [M1, N1] refers to the three feature map pairs [P1, P5], [P1, P6], and [P1, P7]. The following is the specific feature alignment process:

[0041] The input feature map [M1, N1] with the number of channels C is passed through a convolution layer to obtain the feature map [M2, N2]. [M2, N2] is spliced ​​in the second dimension to obtain the feature map F1 with the number of channels 4C. Then, a convolution layer is used to reduce the number of channels of the feature map F1 from 4C to 2C, and the feature map F2 is obtained. In order to distinguish different spatial positions, a 1x1 convolution is used to spatially filter F2, and a SoftMax function is used to obtain the feature map F3 with the number of channels 2. In order to weight different feature channels, a The 1x1 convolution performs channel filtering on F2 and passes it through the Sigmoid function to obtain the feature map F4; the first channel of the feature map F3 is multiplied by the element at the corresponding position of the feature map F4 to obtain the feature map F5; the remaining channels of the feature map F3 are multiplied by the elements at the corresponding position of the feature map F4 to obtain the feature map F6; the feature map F5 is weightedly fused with the input feature map P1, and the feature map F6 is weightedly fused with the input feature map P5 to obtain the feature map [M3, N3]; the feature maps M3 and N3 are concatenated in the first dimension to obtain the feature map P'5;

[0042] According to the above steps, the feature maps [P1, P5], [P1, P6], and [P1, P7] are respectively converted into new feature maps P'5, P'6, and P'7 in the feature alignment module; P1, P'5, P'6, and P'7 are spliced ​​and passed through a multi-layer convolutional segmentation head to obtain the final detection result map.

[0043] The present invention also provides a road crack detection system based on a cross-attention guided feature alignment network, comprising the following modules:

[0044] The data set acquisition module is used to collect road crack images as a data set and divide them into a training set and a test set;

[0045] A detection model construction module is used to build an automated crack detection model consisting of an encoder and a decoder. The encoder includes weight-sharing collaborative network modules encoder1 and encoder2, and a cross-layer interaction module. The decoder includes a multi-scale feature fusion and feature alignment module.

[0046] The weight-sharing collaborative network encoder1 and encoder2 each have k stages, each of which includes a multi-scale image block embedding layer and a multi-head convolutional self-attention layer. The crack image is input to obtain feature maps of different scales in k stages respectively;

[0047] The cross-layer interaction module first reshapes and swaps the dimensions of the feature maps of different scales in the same stage of encoder 1 and encoder 2, then normalizes and globally averages the obtained tensors, and finally uses the self-attention mechanism to extract more comprehensive semantic information of crack features, thereby improving the network's multi-scale feature expression capability.

[0048] The multi-scale feature fusion module adopts a dual progressive fusion strategy of reconstruction and upsampling blocks to fuse the extracted features containing rich multi-scale semantic information;

[0049] The input of the feature alignment module is the output image of the cross-layer interaction module. The purpose is to combine the context information in the image to ensure the scale and semantic consistency between features at different levels or branches, thereby improving the detection ability of small cracks.

[0050] The road crack detection module is used to train the automatic crack detection model using the training set and realize road crack detection using the trained model.

[0051] Overall, the technical solution conceived by this invention achieves the following advantages compared to existing technologies: Based on a deep convolutional neural network, an automated road crack detection model consisting of an encoder and decoder is constructed, which effectively extracts and integrates multi-scale feature information from road crack images, significantly improving the accuracy of road crack detection. Ultimately, simply inputting road crack images into the trained model allows for automatic crack detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a schematic diagram of a process provided by an embodiment of the present invention;

[0053] Figure 2 Schematic diagram of an automated road crack detection model provided by an embodiment of the present invention;

[0054] Figure 3 is a structural diagram of a feature extraction module provided by an embodiment of the present invention;

[0055] Figure 4 This is a structural diagram of a cross-layer interaction module provided by an embodiment of the present invention;

[0056] Figure 5 This is a structural diagram of a feature interaction module provided by an embodiment of the present invention;

[0057] Figure 6 3 is a comparison chart of detection using different methods provided by the embodiments of the present invention. DETAILED DESCRIPTION

[0058] The present invention will be further described in detail below with reference to the accompanying drawings and examples.

[0059] This embodiment of the present invention employs a feature alignment network based on cross-attention guidance to extract and integrate multi-scale features of road cracks. First, a dual-branch visual encoder model with the same architecture but different patch sizes (one large and one small) is used to extract multi-level damage features. A cross-layer interaction module establishes interaction between the corresponding layers of the two branches, combining their respective feature extraction capabilities and contextual understanding. Second, a feature alignment block is used to semantically and spatially align features at different levels or branches. This significantly improves the model's perception of crack areas, reduces background interference, and enables more accurate damage detection and segmentation. Finally, a multi-layer convolutional segmentation head is used to obtain high-resolution feature maps. The performance of the present invention is evaluated using eight metrics: precision, recall, a comprehensive evaluation index (F1-score), intersection over union (IOU), intersection over no cracks (IOU_0), intersection over cracks (IOU_1), overall accuracy (OA), and kappa coefficient (Kappa). The present invention is further described in detail below with reference to the accompanying figures and examples.

[0060] Figure 1 FIG. 1 is a flow chart of a method for automatically detecting road cracks according to an embodiment of the present invention, which specifically includes the following steps:

[0061] Step 1: Use artificially created road crack images as a dataset and create a binary label image based on the crack images in the dataset. Each pixel in the label image represents a category, either crack or no crack.

[0062] In some optional implementation schemes, road crack images collected by a high-definition camera on a vehicle are used as a data set, and a binary label image is produced based on the crack images in the data set, wherein each pixel in the label image represents a category, namely crack or background;

[0063] Constructing a reasonable road crack detection dataset can effectively improve the detection accuracy of the model. In the experiment of the embodiment of the present invention, the CRACK500 dataset was used, which contains 500 images with a resolution of about 2000×1500 from the dataset. These images were taken with a mobile phone from the main campus of Temple University. Each image is divided into 16 non-overlapping crack areas with a resolution of 256×256, and due to the limitation of computing resources, only areas containing more than 1000 crack pixels are retained, which contains a total of 3368 images. The present invention uses professional computer vision annotation software to perform pixel-level annotation on road crack images. A binary label image is produced based on the crack image in the dataset. Among them, the class labels in the road crack image are represented by 0 and 1, 0 represents no crack (can be displayed as black), and 1 represents a crack area (can be displayed as white).

[0064] After the above processing, the road crack images and their corresponding label images are obtained to form a road crack automatic detection image dataset. In this dataset, the training set (a total of 2694 images) and the test set (a total of 674 images) are divided into a training set and a test set according to the ratio of 8:2.

[0065] Step 2: Build an automated road crack detection model: Build a cross-attention-guided feature alignment network consisting of an encoder and a decoder as the automated crack detection model;

[0066] like Figure 2 As shown, the automated road crack detection model in this embodiment of the present invention consists of two main modules: an encoder and a decoder. The encoder includes weight-sharing collaborative network modules encoder1 and encoder2, as well as a cross-layer interaction module. The decoder includes multi-scale feature fusion and feature alignment modules. The encoder is responsible for extracting multi-scale feature information from the input image. The decoder performs progressive interactive fusion of the extracted multi-scale features and calculates a probability loss based on contextual differential information, continuously improving the resulting image's approximation to the ground truth.

[0067] In this embodiment, the weight-shared collaborative network in the encoder uses a dual-branch visual encoder model with the same structure but different patch sizes (one large patch and one small patch) to extract multi-level damage features. The cross-layer interaction module in the encoder establishes interaction between the corresponding layers of the two branches, combining their respective feature extraction capabilities and context understanding, and promoting information transfer, feature fusion, and gradient propagation. It can improve the perception, expression, and performance of the model, thereby more accurately detecting road cracks. The feature alignment module in the decoder uses multi-scale feature alignment technology to fuse the multi-stage output features extracted from the encoder. The purpose is to improve the consistency and fusibility of the features by adjusting and matching features of different levels, different modalities, and different scales, thereby improving the performance and accuracy of the model.

[0068] like Figure 2 As shown, taking K=4 as an example, the weight-shared collaborative network in step 2 performs feature extraction operations on the input road crack image. Given an input original image F1 of size H×W×3, first use a backbone block consisting of two 3×3 convolutional layers to adjust the image size and number of channels to obtain the input features of the multi-scale image block embedding layer in stage1. The number of channels of the first convolutional layer is C1 / 2, and the number of channels of the second convolutional layer is C1, and their strides are both 2. The size of the feature map generated by the two convolution operations is H / 4×W / 4×C1, where C1 is the channel size of stage1. Each convolution is followed by Batch Normalization and Hardswish activation function. As shown Figure 3 As shown in Figure 2, the feature maps obtained from the above backbone blocks are fed into the multi-scale image patch embedding layer of stage 1, which utilizes both fine-grained and coarse-grained visual labels at the same feature level. To this end, convolution operations with overlapping patches are used. Specifically, for the 2D reshaped output feature map (i.e., label map) from the previous stage As the input of stage i, we learn a function F k×k (·), X i Mapped to have channel size C i New mark F k×k (X i ), where F(·) is a 2D convolution operation with kernel size (i.e., patch size) of k×k, stride s, and padding p. The output 2D label map With the following height and width: The multi-scale image block embedding layer adjusts the length of the labeled sequence by changing the stride and padding, and outputs features of the same size (i.e., resolution) with different label sizes. In this embodiment, three features of different sizes are output. Finally, the labeled feature maps of different scales obtained by the multi-scale image block embedding layer are independently input into each multi-scale self-attention head of the multi-head convolutional self-attention layer. In each layer, the input is first normalized by Layer Norm, then self-attention is calculated by Factorized MHSA, and then the original input is added to the attention output by residual connection (Add), and then Layer Norm normalization is performed. After that, nonlinear transformation is performed by FFN, residual connection is performed again, and finally Layer Norm normalization is performed again. Such layer stacking allows the model to gradually learn higher-level features and semantic representations, which helps to better capture the information in the input sequence. The obtained features are aggregated by multi-scale self-attention heads, so as to achieve fine and coarse feature representations at the same feature level and obtain global feature maps of different scales. The features at the last scale output by the multi-scale image patch embedding layer are fed into the convolutional local feature block. In the convolutional local feature block of the multi-scale self-attention layer, a residual bottleneck block consisting of a 1×1 convolution, a 3×3 depthwise convolution, and a 1×1 convolution is used to obtain a local feature map. Finally, the multi-scale feature interaction block (including a merge operation and a 1×1 convolution) aggregates the local and global features to produce the output feature map of stage 1. This cycle repeats to obtain the output feature maps of stages 2, 3, and 4.

[0069] After the above operations, the final feature maps obtained by stages 1, 2, 3, and 4 in encoder 1 have sizes of 64×64, 32×32, 16×16, and 8×8, respectively. Their resolutions are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input image, respectively. Furthermore, the number of channels in these four feature maps is 64, 128, 256, and 512, respectively. Similarly, the original image is also input into encoder 2, and the same feature extraction operations as encoder 1 are performed. Finally, the four stages produce feature maps of sizes 64×64, 32×32, 16×16, and 8×8, with numbers of channels 128, 256, 512, and 1024, respectively.

[0070] like Figure 4As shown, the cross-layer interaction module in step S2 performs interactive operations on the eight outputs of the collaborative network. The input feature matrices, whose shapes are [B, C, H, W] and [B, 2C, H, W], are reshaped, converting them from 4D tensors to 3D tensors. The second and third dimensions are swapped, resulting in tensors e and r, respectively, where B is the batch size, C is the number of channels, H is the height of the feature map, and W is the width of the feature map. [B, C, H, W] represents the feature maps obtained in each of the four stages of encoder 1, and [B, 2C, H, W] represents the feature maps obtained in each of the four stages of encoder 2. The number of channels between feature maps in the same stage of encoder 1 and encoder 2 is doubled. Tensors e and r are simultaneously normalized and globally average pooled, resulting in feature vectors of length 1, denoted as e_t and r_t. Map e_t and r_t to each other's dimensions, and insert a dimension of length 1 along the second dimension. Concatenate r and e_t, and e and r_t along the second dimension, respectively. Then, pass the concatenated feature tensors simultaneously into the Transformer encoder for their respective cross-attention calculations. Finally, four feature maps (denoted as P1, P2, P3, and P4) are obtained, with sizes of 64×64, 32×32, 16×16, and 8×8, respectively, and with 128, 256, 512, and 1024 channels, respectively.

[0071] In some optional implementation schemes, in step S2, the cross-attention calculation formula of the cross-layer interaction module is:

[0072]

[0073] in, represents the query vector matrix, where m represents the number of query vectors, d k represents the dimension of the key vector; represents the key vector matrix, where n represents the number of key vectors, d k represents the dimension of the key vector; represents a value vector matrix, where d v Represents the dimension of the key vector; QK T Represents the dot product matrix between the query vector and the key vector, whose dimension is m×n; softmax() represents the softmax function, which is used to normalize the score to a probability value; Used to normalize the dot product to prevent the dot product value from being too large or too small.

[0074] Based on the above feature interaction process, K feature maps of different scales can be obtained by analogy. In the multi-scale feature fusion module, a dual progressive fusion strategy of reconstruction and upsampling blocks is used to fuse these K feature maps. This is a layer-by-layer cascade feature fusion process, which gradually fuses the initial feature map with subsequent feature maps to ultimately obtain a comprehensive global feature representation. Taking K = 4 as an example, the specific implementation of fusion is as follows:

[0075] In the first step, the four feature maps (P1, P2, P3, and P4) obtained by the cross-layer interaction module are concatenated in their respective first dimensions, and then four new feature maps (denoted as A, B, C, and D) are generated through convolution operations.

[0076] In the second step, D is upsampled and then convolved to obtain a new feature map D', and then C and D' are concatenated in the first dimension and then convolved to obtain a feature map D";

[0077] In the third step, D' is upsampled and then convolved to obtain a new feature map D'', and then B and D'' are concatenated in the first dimension and then convolved to obtain the feature map B';

[0078] The fourth step is to upsample B' and then convolve it to get the new feature map B", then concatenate A and B' in the first dimension and then convolve it to get the feature map A';

[0079] In the fifth step, a convolutional layer with a convolution kernel size of 3 is used to reduce the number of channels of the feature maps D', B', and A' to 256. At the same time, the feature maps after dimensionality reduction are upsampled and their sizes are all adjusted to 64×64 to obtain three new feature maps P5, P6, and P7;

[0080] Therefore, after the above five-step multi-scale fusion operation, four new feature maps are obtained: P1, P5, P6, and P7. These four new feature maps are the input of the feature alignment module.

[0081] like Figure 5 As shown in Figure 2, the feature alignment module in step S2 performs feature alignment on the four outputs of the multi-scale feature fusion module. The process is to align P1 with P5, P6, and P7 respectively to obtain three new feature maps, and then merge P1 and these three new feature maps to obtain the final output feature map of the feature alignment module. [M1, N1] refers to the three feature map pairs [P1, P5], [P1, P6], and [P1, P7]. The following is the specific feature alignment process:

[0082] The input feature map [M1, N1] with C channels is passed through a convolution layer with a kernel size of 1 to obtain the feature map [M2, N2], increasing the number of channels to 512. [M2, N2] is concatenated in the second dimension to obtain the feature map F1 with 4C channels. A convolution with a kernel size of 1 is then performed to reduce the number of channels in feature map F1 from 4C to 2C, resulting in feature map F2. To distinguish different spatial locations, a 1x1 convolution is performed on F2, and a softmax function is applied to obtain a feature map F3 with 2 channels. To weight different feature channels, a 1x1 convolution is performed on F2, and a sigmoid function is applied to obtain a feature map F4 with 256 channels. The first channel of feature map F3 is multiplied by the element at the corresponding position in feature map F4 to obtain feature map F5. The remaining channels of feature map F3 are multiplied by the elements at the corresponding positions in feature map F4 to obtain feature map F6. Feature map F5 is weightedly fused with input feature map P1, and feature map F6 is weightedly fused with input feature map P5 to obtain feature map [M3, N3]. Feature maps M3 and N3 are concatenated along the first dimension to obtain feature map P'5. Following these steps, feature maps [P1, P5], [P1, P6], and [P1, P7] are used in the feature alignment module to generate new feature maps P'5, P'6, and P'7, respectively. P1, P'5, P'6, and P'7 are concatenated and passed through a multi-layer convolutional segmentation head to obtain the final detection result map.

[0083] Step 3: Use the training set in step 1 to train the road crack automatic detection model in step 2, use the trained model to detect cracks, and finally evaluate the detection results using the crack automatic detection model evaluation index;

[0084] The network structure proposed in this invention was trained on the CRACK500 dataset constructed in step S1 to obtain model weights for model evaluation. The training process was based on the PyTorch deep learning framework, using Ubuntu 20.04 as the software environment and a 3090 graphics card with 24GB of video memory. The batch size was set to 4, and the total number of training epochs was 300. Each input consisted of one image: the source road crack image. After each training session, a test was performed. During network training, the network continuously learned crack information from the real road crack images. The training loop continued until 300 epochs were reached, at which point training concluded.

[0085] The precision (Precision), recall (Recall), comprehensive evaluation index (F1-score) intersection (IOU), no crack intersection (IOU_0), overall accuracy (OA), crack intersection (IOU_1), Kappa coefficient (Kappa) are selected as evaluation indicators, and the calculation formula of the evaluation indicators is as follows:

[0086]

[0087]

[0088]

[0089]

[0090]

[0091]

[0092]

[0093] In order to verify the performance of the building automation detection model proposed in this invention, the present invention provides the final experimental results. Figure 6 is a visual comparison chart of various methods, and Table 1 is the quantitative indicators of various methods.

[0094] in, Figure 6 Figure 2 shows the road crack detection results obtained by various methods. (a) is the source road crack image, (b) is the ground truth (GT) crack image, and (c)-(j) are the detection results of different methods. By comparing the ground truth crack images, black represents crack-free areas and white represents cracked areas.

[0095] Table 1: Accuracy of various methods in building detection on the CRACK500 dataset

[0096]

[0097] Please note that all indicators are expressed in percentages, and the larger the value, the better the effect. For easier viewing, the best indicator results are highlighted in bold.

[0098] The embodiment of the present invention further provides a road crack detection system based on a cross-attention guided feature alignment network, comprising the following modules:

[0099] The data set acquisition module is used to collect road crack images as a data set and divide them into a training set and a test set;

[0100] A detection model construction module is used to build an automated crack detection model consisting of an encoder and a decoder. The encoder includes weight-sharing collaborative network modules encoder1 and encoder2, and a cross-layer interaction module. The decoder includes a multi-scale feature fusion and feature alignment module.

[0101] The weight-sharing collaborative network encoder1 and encoder2 each have k stages, each of which includes a multi-scale image block embedding layer and a multi-head convolutional self-attention layer. The crack image is input to obtain feature maps of different scales in k stages respectively;

[0102] The cross-layer interaction module first reshapes and swaps the dimensions of the feature maps of different scales in the same stage of encoder 1 and encoder 2, then normalizes and globally averages the obtained tensors, and finally uses the self-attention mechanism to extract more comprehensive semantic information of crack features, thereby improving the network's multi-scale feature expression capability.

[0103] The multi-scale feature fusion module adopts a dual progressive fusion strategy of reconstruction and upsampling blocks to fuse the extracted features containing rich multi-scale semantic information;

[0104] The input of the feature alignment module is the output image of the cross-layer interaction module. The purpose is to combine the context information in the image to ensure the scale and semantic consistency between features at different levels or branches, thereby improving the detection ability of small cracks.

[0105] The road crack detection module is used to train the automatic crack detection model using the training set and realize road crack detection using the trained model.

[0106] The specific implementation of each module corresponds to each step and is not described in detail in the present invention.

[0107] It should be pointed out that, according to the needs of implementation, the various steps / components described in this application can be split into more steps / components, or two or more steps / components or partial operations of steps / components can be combined into new steps / components to achieve the purpose of the present invention.

[0108] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the invention are included in the scope of protection of the present invention.

Claims

1. A road crack detection method based on cross-attention guided feature alignment network, characterized in that The following steps are involved: Step 1: Collect road crack images as a dataset and divide them into a training set and a test set; Step 2: Build an automated crack detection model consisting of an encoder and a decoder. The encoder includes weight-sharing collaborative network modules encoder1 and encoder2, and a cross-layer interaction module. The decoder includes a multi-scale feature fusion module and a feature alignment module. The weight-sharing collaborative network encoder1 and encoder2 each have k stages, each of which includes a multi-scale image block embedding layer and a multi-head convolutional self-attention layer. The crack image is input to obtain feature maps of different scales in k stages respectively; The cross-layer interaction module first reshapes and swaps the dimensions of the feature maps of different scales in the same stage of encoder 1 and encoder 2, then normalizes and globally averages the obtained tensors, and finally uses the self-attention mechanism to extract more comprehensive semantic information of crack features. The specific implementation of the cross-layer interaction module in step 2 includes the following sub-steps: 221) reshape the input feature matrices whose feature shapes are [B, C, H, W] and [B, 2C, H, W], respectively, converting them from 4D tensors to 3D tensors and swapping their second and third dimensions to obtain tensors e and r, respectively; [B, C, H, W] represents the collective name of the feature maps obtained at each stage in encoder1, and [B, 2C, H, W] represents the collective name of the feature maps obtained at each stage in encoder2. The number of channels between the feature maps of encoder1 and encoder at the same stage is twice that of the other. 222) Perform normalization and global average pooling operations on tensors e and r simultaneously to obtain feature vectors of length 1, denoted as e_t and r_t; 223) Map e_t and r_t to each other's dimensions, and insert a dimension of length 1 on the second dimension; 224) Concatenate r and e_t, e and r_t respectively in the second dimension, and then pass the concatenated feature tensors into the Transformer encoder for their respective cross-attention calculations, and finally obtain multiple feature maps, denoted as P1, P2, ..., P k ; The multi-scale feature fusion module adopts a dual progressive fusion strategy of reconstruction and upsampling blocks to fuse the extracted features containing rich multi-scale semantic information; The feature alignment module combines the context information in the image to perform feature alignment on the output image of the multi-scale feature fusion module to ensure the scale and semantic consistency between features at different levels or branches; Step 3: Use the training set in step 1 to train the automatic crack detection model in step 2, and use the trained model to realize road crack detection.

2. The method according to claim 1, characterized in that Step 1 includes: 11) Using road crack images collected by a high-definition camera on a vehicle as a dataset, a binary label image is created based on the crack images in the dataset, where each pixel in the label image represents a category, either crack or background; 12) The road crack images and their corresponding binary label images are combined into a road crack automatic detection image dataset, and the dataset is divided into a training set and a test set according to a certain ratio.

3. The method according to claim 1, wherein: The specific implementation of the weight-sharing collaborative network processing process in step 2 includes the following sub-steps: 211) Given an input original image F1, first use a backbone block consisting of two convolutional layers to adjust the image size and number of channels to obtain the input features of the multi-scale image block embedding layer in the first stage; 212) Input features are fed into encoder1 and encoder2 simultaneously for feature extraction. Each stage includes a multi-scale image block embedding layer and a multi-head convolutional self-attention layer to extract crack information. 213) The multi-scale image patch embedding layer utilizes fine-grained and coarse-grained visual markers at the same feature level using convolution operations with overlapping patches; specifically, for the 2D reshaped output feature map from the previous stage As the input of stage i, we first learn a function F k×k (·), X i Mapped to have channel size C i New mark F k×k (X i ), where F(·) is a 2D convolution operation with a kernel size of k×k, a stride of s, and a padding of p; the output 2D label map With the following height and width: The multi-scale image patch embedding layer adjusts the length of the labeled sequence by changing the stride and padding, outputting features of the same resolution size with different label sizes; 214) The labeled feature maps of different scales obtained by the multi-scale image block embedding layer are respectively input into each self-attention layer of the multi-head convolutional self-attention layer to obtain global feature maps of different scales; the features of the last scale among the features output by the multi-scale image block embedding layer are input into the convolutional local feature block to obtain a local feature map; finally, the local features and global features are aggregated using the multi-scale feature interaction block, and the output feature map of the first stage is obtained through a convolution operation; Each stage of 215) is processed according to steps 213)-215), and encoder1 and encoder2 respectively output multiple feature maps of different scales.

4. The method according to claim 3, wherein: The number of channels of the first convolutional layer in the backbone block is C1 / 2, and the number of channels of the second convolutional layer is C1. The size of the feature map generated by the two convolution operations is H / 4×W / 4×C1, where C1 is the channel size of the first stage. Each convolution is followed by Batch Normalization and Hardswish activation function.

5. The method according to claim 3, wherein: The processing process of the multi-head convolutional self-attention layer is as follows: the input is first normalized by Layer Norm, then self-attention calculation is performed through Factorized MHSA, and then the original input and attention output are added through residual connection, and then LayerNorm normalization is performed. After that, nonlinear transformation is performed through FFN, and then residual connection is performed again, and finally LayerNorm normalization is performed again; the convolutional local feature block is implemented using a residual bottleneck block consisting of 1×1 convolution, 3×3 depth convolution and 1×1 convolution.

6. The method according to claim 1, wherein: In step S2, the cross-attention calculation formula of the cross-layer interaction module is: in, represents the query vector matrix, where m represents the number of query vectors, d k represents the dimension of the key vector; represents the key vector matrix, where n represents the number of key vectors, d k represents the dimension of the key vector; represents a value vector matrix, where d v Represents the dimension of the key vector; QK T Represents the dot product matrix between the query vector and the key vector, whose dimension is m×n; softmax() represents the softmax function, which is used to normalize the score to a probability value; Used to normalize the dot product to prevent the dot product value from being too large or too small.

7. The method according to claim 1, wherein: The cross-layer interaction module obtains K feature maps of different scales. When K = 4, the multi-scale feature fusion module implements the fusion processing of K feature maps through the following steps; In the first step, the four feature maps P1, P2, P3, and P4 obtained by the cross-layer interaction module are concatenated in their respective first dimensions, and then four new feature maps are generated through convolution operations, which are recorded as A, B, C, and D. In the second step, D is upsampled and then convolved to obtain a new feature map D', and then C and D' are concatenated in the first dimension and then convolved to obtain a feature map D"; In the third step, D' is upsampled and then convolved to obtain a new feature map D'', and then B and D'' are concatenated in the first dimension and then convolved to obtain the feature map B'; The fourth step is to upsample B' and then convolve it to get the new feature map B", then concatenate A and B' in the first dimension and then convolve them to get the feature map A'; In the fifth step, a convolutional layer is used to reduce the number of channels of the feature maps D'', B', and A', and the reduced feature maps are upsampled to obtain three new feature maps P5, P6, and P7; Therefore, after the above five-step multi-scale fusion operation, four new feature maps are obtained: P1, P5, P6, and P7. These four new feature maps are the input of the feature alignment module.

8. The method according to claim 7, wherein: The feature alignment module in step S2 performs feature alignment on the four outputs of the multi-scale feature fusion module. The process is to align P1 with P5, P6, and P7 respectively to obtain three new feature maps. Then, P1 and these three new feature maps are merged to obtain the final output feature map of the feature alignment module. [M1, N1] refers to the three feature map pairs [P1, P5], [P1, P6], and [P1, P7]. The following is the specific feature alignment process: The input feature map [M1, N1] with the number of channels C is passed through a convolution layer to obtain the feature map [M2, N2]. [M2, N2] is spliced ​​in the second dimension to obtain the feature map F1 with the number of channels 4C. Then, a convolution layer is used to reduce the number of channels of the feature map F1 from 4C to 2C, and the feature map F2 is obtained. In order to distinguish different spatial positions, a 1x1 convolution is used to spatially filter F2, and a SoftMax function is used to obtain the feature map F3 with the number of channels 2. In order to weight different feature channels, a 1x1 convolution performs channel screening on F2 and passes it through the Sigmoid function to obtain feature map F4; the first channel of feature map F3 is multiplied by the element at the corresponding position of feature map F4 to obtain feature map F5; the remaining channels of feature map F3 are multiplied by the elements at the corresponding position of feature map F4 to obtain feature map F6; feature map F5 is weightedly fused with input feature map P1, and feature map F6 is weightedly fused with input feature map P5 to obtain feature map [M3, N3]; feature maps M3 and N3 are concatenated in the first dimension to obtain feature map P5'; According to the above steps, the feature maps [P1, P5], [P1, P6], and [P1, P7] are respectively converted into new feature maps P5', P6', and P7' in the feature alignment module; P1, P5', P6', and P7' are spliced ​​and passed through a multi-layer convolutional segmentation head to obtain the final detection result map.

9. A road crack detection system based on cross-attention guided feature alignment network, characterized in that Includes the following modules: The data set acquisition module is used to collect road crack images as a data set and divide them into a training set and a test set; A detection model construction module is used to build an automated crack detection model consisting of an encoder and a decoder. The encoder includes weight-sharing collaborative network modules encoder1 and encoder2, and a cross-layer interaction module. The decoder includes a multi-scale feature fusion module and a feature alignment module. The weight-sharing collaborative network encoder1 and encoder2 each have k stages, each of which includes a multi-scale image block embedding layer and a multi-head convolutional self-attention layer. The crack image is input to obtain feature maps of different scales in k stages respectively; The cross-layer interaction module first reshapes and swaps the dimensions of the feature maps of different scales in the same stage of encoder 1 and encoder 2, then normalizes and globally averages the obtained tensors, and finally uses the self-attention mechanism to extract more comprehensive semantic information of crack features. The specific implementation of the cross-layer interaction module includes the following sub-steps: 221) reshape the input feature matrices whose feature shapes are [B, C, H, W] and [B, 2C, H, W], respectively, converting them from 4D tensors to 3D tensors and swapping their second and third dimensions to obtain tensors e and r, respectively; [B, C, H, W] represents the collective name of the feature maps obtained at each stage in encoder1, and [B, 2C, H, W] represents the collective name of the feature maps obtained at each stage in encoder2. The number of channels between the feature maps of encoder1 and encoder at the same stage is twice that of the other. 222) Perform normalization and global average pooling operations on tensors e and r simultaneously to obtain feature vectors of length 1, denoted as e_t and r_t; 223) Map e_t and r_t to each other's dimensions, and insert a dimension of length 1 on the second dimension; 224) Concatenate r and e_t, e and r_t respectively in the second dimension, and then pass the concatenated feature tensors into the Transformer encoder for their respective cross-attention calculations, and finally obtain multiple feature maps, denoted as P1, P2, ..., P k ; The multi-scale feature fusion module adopts a dual progressive fusion strategy of reconstruction and upsampling blocks to fuse the extracted features containing rich multi-scale semantic information; The feature alignment module combines the context information in the image to perform feature alignment on the output image of the multi-scale feature fusion module to ensure the scale and semantic consistency between features at different levels or branches; The road crack detection module is used to train the automatic crack detection model using the training set and realize road crack detection using the trained model.

Images