Bridge crack detection method based on multi-scale feature fusion and multi-layer attention
Through multi-scale feature fusion and multi-layer attention detection methods, the problems of low efficiency and high error detection rate of traditional detection methods are solved, efficient and accurate bridge crack detection is achieved, and the timeliness and robustness of detection is improved.
Patent Information
- Application Number
- CN202411077414.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-11-12
AI Technical Summary
Traditional manual detection methods lead to low efficiency, poor timeliness, high error detection rate, insufficient robustness and generalization, which cannot meet the rapid development needs of modern industry.
Using a bridge crack detection method based on multi-scale feature fusion and multi-layer attention, a detection network model containing encoder and decoder is constructed, and a Mobilenetv2 network and Bottleneck Transformer module are used to combine a hollow space pyramid pool with nested multi-layer attention. The integrated structure and multi-scale feature fusion module can improve the feature extraction and segmentation effect.
It significantly improves the accuracy and generalization ability of bridge crack detection, enhances the effect of extracting crack features, improves the practicality and adaptability of detection, reduces the error detection rate, and meets the needs of efficient and accurate detection.
Smart Images

Figure CN118941542A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a crack detection technology, and in particular to a lightweight bridge crack detection method based on a multi-layer attention mechanism and multi-scale feature fusion. Background Art
[0002] The manual detection method of bridge cracks is inefficient and can no longer meet the needs of the rapid development of modern industry, so it has gradually lost its applicability. Although the technology based on morphological bridge crack detection has been widely used in the field of crack detection due to its simplicity and efficiency, it mainly uses traditional image processing methods to analyze and identify bridge crack images, which may cause the following problems:
[0003] 1) Using traditional manual detection methods, the crack detection cycle is often as long as 3 to 6 months, which seriously restricts the maintenance decision-making at the cracks and has lost its timeliness to a certain extent; at the same time, the use of manual crack detection is affected by factors such as insufficient automation of detection technology and high degree of subjective judgment by inspectors using the naked eye, and its accuracy and stability are often also affected.
[0004] 2) Inconsistent parameters: Manually setting the detection parameters results in different parameters required for crack images in different backgrounds or environments, which increases the complexity of the operation and makes the process cumbersome.
[0005] 3) False detection and accuracy issues: Since the morphological method is sensitive to changes in image illumination, noise, etc., it is difficult to accurately segment the crack area in complex backgrounds and crack diversity scenarios, such as blurred edges, small crack sizes, and poor continuity, resulting in a high false detection rate and low accuracy, causing the detection results to fail to meet the requirements.
[0006] 4) Robustness or generalization issues: Pavement crack detection requires the identification and processing of different types of cracks. Therefore, the algorithm needs to have strong robustness and generalization capabilities and be able to adapt to different environments and different crack types. Summary of the invention
[0007] Aiming at the problems of false detection, insufficient precision, poor robustness or generalization in crack image recognition using traditional image processing methods, a bridge crack detection method based on multi-scale feature fusion and multi-layer attention was proposed.
[0008] The technical solution of the present invention is:
[0009] A bridge crack detection method based on multi-scale feature fusion and multi-layer attention includes the following steps:
[0010] Step 1) Collect crack image sets under different environments to obtain more diverse and widely covered samples, and divide the collected samples into two parts: training set and test set;
[0011] Step 2) Construct a bridge crack detection network model based on efficient feature fusion and multi-layer attention; the model includes an encoder that converts the rich feature information data of the bridge crack image into a specific encoding form in order to more effectively represent and process the rich feature information data of the bridge crack image during transmission or storage, and a decoder that reconverts the data encoded by the encoder into the original crack image data. The two parts are constructed in a serial manner; for the feature extraction stage, the model structure includes multiple convolutional layers, each of which is connected to a batch normalization layer and an activation function layer. The activation function uses the linear rectified unit ReLU function, and the convolution kernel and weight parameters are initialized using a Gaussian random function with a variance of 0.01; in the encoder part, the model uses the Mobilenetv2 network module as the backbone network to keep the model lightweight to a certain extent and ensure the feature extraction capability, and uses a hollow space pyramid pooling structure with nested multi-layer attention to accurately extract the rich feature information data in the bridge crack image; the decoder part uses a multi-scale feature fusion module and a combination of Bottleneck Transformer’s high- and low-level feature adaptive weighted feature fusion module structure. The adaptive weighted feature fusion module strategically fuses the high-level features extracted from the encoder end with the low-level features extracted by the multi-scale feature extraction module to enhance the feature extraction effect, thereby enhancing the segmentation effect of bridge cracks. Finally, the obtained image is upsampled 4 times to obtain a prediction map that matches the input image resolution.
[0012] Step 3) training the network; first, the training sample set is sent to the model constructed in step 2) in batches, and the training samples pass through the encoder and decoder of the model in turn; then, the momentum parameter, weight decay, batch size, learning rate, and training epoch of the model training are adjusted to achieve a good training effect; finally, the three mean square errors are linearly combined as the loss function, and the abundance matrix is constrained by L2 regularization, the model is iteratively trained by back propagation, and the model parameters are adjusted. In each iteration, the end member matrix is fine-tuned using the multiplication rule until convergence;
[0013] Step 4) testing the model; sending the test sample set to the model trained in step 3) to obtain the corresponding test generation result;
[0014] Step 5) Evaluate the performance of the algorithm; Based on the three improvements of the proposed model, the ablation experiment and the overall comparative experiment are performed to obtain the bridge crack detection results. At the same time, the effectiveness of the algorithm is measured by relevant evaluation indicators such as MIoU, IoU, and F1 score. The loss curve and MIoU curve of the improved model are also recorded in the experiment to reflect the experimental effect;
[0015] Among them, the structure of the high-level and low-level feature adaptive weighted feature fusion module combined with Bottleneck Transformer in step 2) is specifically as follows:
[0016] First, the high-level features output by the encoder are linearly upsampled by 4 times to match the resolution of the low-level feature map, generating the upsampled feature map F 4H , as shown in formula (1); then, the upsampled high-level features and low-level features are concatenated in the channel dimension to obtain F L-4H , as shown in formula (2); then, as shown in formulas (3) and (4), the concatenated features are input into the Bottleneck Transformer module and then through the Conv 1×1 And ReLU activation function generates spatial feature map W R ;
[0017]
[0018] W R =δ(W L-4H ) (4)
[0019] where F L represents the low-level feature map, F H represents a high-level feature map, represents feature connection, Represents bilinear interpolation;
[0020] For low-level features F L and spatial feature map W R Perform adaptive weighted feature fusion to obtain W W-Add , as shown in formula (5); this can more effectively locate the target object and use spatial details to obtain finer shape and boundary information; then, the weighted features and high-level features are directly merged; the resulting feature map is then subjected to a convolution operation to obtain a refined and concentrated fusion feature as shown in formula (6); finally, this output is upsampled to match the resolution of the input image to generate a prediction map;
[0021] W W-Add =Σ(W R ,F L )=ω1W R +ω2F L(5)
[0022] ω1+ω2=1 (6)
[0023]
[0024] in Represents Conv 1×1 operate, Represents Conv 3×3 Operation, f Trans represents the Bottleneck Transformer module, δ represents the ReLU activation function, Σ represents the adaptive weighted feature fusion operation, and ω1, ω2 represent the weighting coefficients, and their relationship is shown in Equation (6);
[0025] Finally, the weighted features and high-level features are directly concatenated; the concatenated feature map undergoes a 3×3 convolution operation to generate a refined and concentrated fusion feature, as shown in formula (7); then, this output is upsampled to match the resolution of the input image to generate a prediction map;
[0026] Among them, the hollow space pyramid pooling structure with nested multi-layer attention is as follows:
[0027] The structure introduces channel prior convolutional attention CPCA, which effectively integrates global context by connecting multiple layers of dilated convolution and pooling structures in series to form parallel branches; secondly, CPCA dynamically allocates attention weights in channels and space to reduce redundant information. This combination improves the model's understanding of the semantic structure of the image and enables the model to focus more accurately on key semantic areas and enhance edge detail processing; then, the strategy of CPCA dynamically allocating attention weights in channels and space simplifies operations and avoids computational overhead; the structure further introduces a coordinate attention mechanism to integrate information from parallel branches and enhance the model's understanding of target location and context; ultimately, these innovations aim to improve the accuracy of image segmentation by emphasizing relevant areas and suppressing irrelevant information;
[0028] Among them, the multi-scale feature fusion module is as follows:
[0029] The multi-scale feature fusion module extracts multi-level features from the encoder input image, but these different levels of features have multiple differences in spatial resolution due to downsampling, so it is necessary to upsample the low-resolution feature map to make it have a similar spatial size as the high-resolution feature map. Then the upsampled low-level features are fused with the high-level features, and finally the fused features are used in subsequent convolution operations to improve the feature extraction performance of the model.
[0030] Furthermore, in step 1), the high-resolution bridge crack images are cropped to obtain crack images with a cropped image size of 448*448, and the processed image set is divided into two parts, a training set and a test set, according to a ratio of 8 to 2.
[0031] Furthermore, in step 3), the training image set divided in step 1) is sent to the network training, and the PyTorch deep learning framework is adopted to train and test the model; the experimental equipment used includes Python3.8, CUDA 11.1, 15vCPU AMD EPYC 7543 32-core processor and RTX 3090GPU with 24GB memory. The stochastic gradient descent optimizer is used for training, the momentum size is set to 0.9, and the weight decay is 0.0001; the batch size is 16, and the number of training rounds is 200; the learning rate is initially set to 0.007 and adjusted using the cosine annealing method; the input image size is fixed to 448×448.
[0032] Furthermore, the Bottleneck Transformer in step 2) is very lightweight in the Transformer field and can avoid long training time. It contains three tensor compressions, two layers of multi-head self-attention, and two tensor expansions. Tensor compression downsamples the image to effectively extract key features; multi-head self-attention captures abstract and low-level features from the image; tensor expansion further processes and aggregates information in the feature map through convolution operations, and upsamples to restore image details. In addition, the model uses jump connections, which not only optimizes the efficiency of image processing, but also reduces model parameters, thereby achieving efficient and accurate image processing.
[0033] The beneficial effects of the present invention are:
[0034] The present invention proposes an adaptive weighted fusion module of high- and low-level features combined with Transformer, which is used to organically fuse high-level features with low-level features, which can effectively improve the detection accuracy and generalization ability of the model. Secondly, the present invention proposes a hollow spatial pyramid pooling structure with nested multi-layer attention. This innovation significantly improves the model's attention to key details and improves the extraction effect of crack features. Finally, the present invention proposes a multi-scale feature fusion module that effectively integrates crack feature information from different scales, which can help retain more low-level semantic information and improve overall performance.
[0035] Together, these innovations improve the performance of our network for bridge crack detection, making it more practical and adaptable. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1This is a structural diagram of an innovative crack detection network based on efficient feature fusion and multi-layer attention of the present invention;
[0037] Figure 2 This is a structural diagram of the high-level and low-level feature adaptive weighted feature fusion module combined with Transformer in the present invention;
[0038] Figure 3 This is a structural diagram of the BottleNeck Transformer of the present invention;
[0039] Figure 4 The hollow space pyramid pooling structure diagram of the present invention nested with multiple layers of attention;
[0040] Figure 5 Schematic diagram of the channel prior convolutional attention (CPCA) of the present invention;
[0041] Figure 6 This is a structural diagram of the coordinate attention (CA) of the present invention;
[0042] Figure 7 It is a structural diagram of the multi-scale feature fusion module of the present invention;
[0043] Figure 8 A comparative visualization diagram of different methods (ablation experiments) based on private datasets of the present invention;
[0044] Fig. 9 A crack image detection diagram of the network model proposed by the present invention;
[0045] Fig.10 is the training loss graph of the model of the present invention;
[0046] Fig.11 This is the MIoU curve of the model of the present invention. DETAILED DESCRIPTION
[0047] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.
[0048] DeepLabv3Plus is widely used by researchers as a commonly used segmentation model. In traditional convolutional neural networks, the use of fixed-size convolution kernels for feature extraction will result in a small receptive field, making it difficult to capture global information in the image. DeepLabv3Plus uses dilated convolution to expand the receptive field. By introducing holes in the convolution kernel, the network can capture local details and global contextual information at the same time, thereby improving the accuracy of semantic segmentation. In addition, DeepLabv3Plus also uses the atrous spatial pyramid pooling structure (ASPP), which further improves the performance of semantic segmentation by extracting features at different scales and fusing features at different scales. The ASPP module can capture object information at different scales and improve the model's ability to detect objects of varying sizes.
[0049] Based on the classic semantic segmentation network DeepLabV3Plus, an advanced bridge crack detection network is proposed, which is a lightweight bridge crack detection method based on multi-layer attention mechanism and multi-scale feature fusion. This network integrates several key innovations to improve the overall performance of the model. Figure 1 As shown, first, a high-level and low-level feature adaptive weighted fusion module combined with Bottleneck Transformer is proposed to strategically fuse high-level features with low-level features, and improve the detection accuracy by fusing bridge crack feature information from different levels. Secondly, a hole spatial pyramid pooling structure with nested multi-layer attention is proposed. The channel prior convolution attention is connected in series at the back end of the dilated convolution and pooling layers to further enhance the feature extraction capability, and coordinate attention is used to integrate channels, enhance the model's understanding of the target location and context, and output relevant features. Finally, we introduced a multi-scale feature fusion module to effectively integrate crack feature information from different scales, which ensures that more low-level semantic information is retained, thereby contributing to the improvement of overall performance. The following is a detailed explanation of the three innovative modules:
[0050] (1) In the decoder part, a high-level and low-level feature adaptive weighted fusion module combined with Bottleneck Transformer is proposed, such as Figure 2 shown.
[0051] This innovative module corresponds to Improvement 1 in Table 1. In the field of semantic segmentation, the Deeplabv3Plus network aims to effectively utilize the semantic information of low-level and high-level features. To achieve this goal, the network adopts a strategy of directly fusing high-level and low-level features. However, the significant background noise in low-level features hinders the effectiveness of simple connections, because the noise may negatively affect the valuable information extracted by the network from the two feature layers, resulting in a decrease in the overall accuracy of crack segmentation. To address these challenges, it becomes crucial to weight the features according to their relative importance, so mechanisms such as multi-head self-attention mechanisms and special feature weighting methods are incorporated into it, which is consistent with the network's processing method, enabling the network to effectively handle long-distance dependencies and capture global contextual information, ensuring that the network prioritizes information that is beneficial to the task. Bottleneck Transformer is very lightweight in the Transformer field to avoid excessive training time. It contains three tensor compressions, two layers of multi-head self-attention, and two tensor expansions. Tensor compression downsamples the image to effectively extract key features; multi-head self-attention captures abstract and low-level features from the image; tensor expansion further processes and aggregates information in the feature map through convolution operations, and upsamples to restore image details. In addition, the model uses jump connections, which not only optimizes the efficiency of image processing, but also reduces model parameters, thereby achieving efficient and accurate image processing.
[0052] The proposed module introduces a strategic selection process between high-level and low-level features. This strategy allows the network to select low-level features relevant to the task and fuse them with high-level features. Through the adaptive weighting and selection process, the network can more cleverly distinguish between background noise and informative features. The ultimate intended goal of this module is to enhance the segmentation effect by effectively combining high-level and low-level features, while solving the interference problem of background noise in low-level features. Specifically, the implementation of the new module follows the following steps:
[0053] First, the high-level features output by the encoder are linearly upsampled by 4 times to match the resolution of the low-level feature map, generating the upsampled feature map F 4H , as shown in formula (1); then, the upsampled high-level features and low-level features are concatenated in the channel dimension to obtain F L-4H , as shown in formula (2); then, as shown in formulas (3) and (4), the concatenated features are input into the Bottleneck Transformer module and then through the Conv 1×1 And ReLU activation function generates spatial feature map W R ;
[0054]
[0055]
[0056] W R =δ(W L-4H ) (4)
[0057] where F L represents the low-level feature map, F H represents a high-level feature map, represents feature connection, Represents bilinear interpolation;
[0058] For low-level features F L and spatial feature map W R Perform adaptive weighted feature fusion to obtain W W-Add , as shown in formula (5); this can more effectively locate the target object and use spatial details to obtain finer shape and boundary information; then, the weighted features and high-level features are directly merged; the resulting feature map is then subjected to a convolution operation to obtain a refined and concentrated fusion feature as shown in formula (6); finally, this output is upsampled to match the resolution of the input image to generate a prediction map;
[0059] W W-Add =Σ(W R ,F L )=ω1W R +ω2F L (5)
[0060] ω1+ω2=1 (6)
[0061]
[0062] in Represents Conv 1×1 operate, Represents Conv 3×3 Operation, f Trans represents the Bottleneck Transformer module, δ represents the ReLU activation function, Σ represents the adaptive weighted feature fusion operation, and ω1, ω2 represent the weighting coefficients, and their relationship is shown in Equation (6);
[0063] Finally, the weighted features and high-level features are directly concatenated; the concatenated feature map undergoes a 3×3 convolution operation to generate a refined and concentrated fusion feature, as shown in formula (7); then, this output is upsampled to match the resolution of the input image to generate a prediction map;
[0064] (2) In the encoder part, a dilated spatial pyramid pooling structure with nested multi-layer attention is proposed.
[0065] This improved module corresponds to Improvement 2 in Table 1. The ASPP (Atrous Spatial Pyramid Pooling) module is a key component of DeepLabV3Plus. ASPP captures multi-scale contextual information in the image through dilated convolutions with different dilation rates to prevent semantic information loss and improve the model's ability to understand the semantic structure. However, the use of multiple dilated convolution layers in ASPP may lead to the extraction and preservation of redundant information, reducing the efficiency of feature extraction. The superposition of multiple layers of complex dilated convolutions is prone to lose the detailed information of the crack edges in the image, resulting in a decrease in segmentation effect. To solve these problems, we propose a nested multi-layer attention atrous spatial pyramid pooling structure, such as Figure 4 shown.
[0066] This structure introduces channel prior convolutional attention (CPCA), such as Figure 5 As shown in , it effectively integrates global context by connecting multiple layers of dilated convolution and pooling structures in series and forming parallel branches. Secondly, CPCA dynamically allocates attention weights in channels and spaces to reduce redundant information. This combination improves the model's ability to understand the semantic structure of the image and enables the model to focus more accurately on key semantic areas and enhance edge detail processing. Then, CPCA dynamically allocates attention weights in channels and spaces. This strategy simplifies the operation and avoids computational overhead. The structure further introduces the coordinate attention mechanism (CA), such as Figure 6 As shown in Figure 2, it is used to integrate information from parallel branches and enhance the model's understanding of target location and context. Ultimately, these innovations aim to improve the accuracy of image segmentation by emphasizing relevant areas and suppressing irrelevant information.
[0067] (3) A multi-scale feature fusion model is proposed in the encoder part.
[0068] This improved module corresponds to Improvement 3 in Table 1. In the semantic segmentation task, the role of the encoder is to extract rich feature information from the input image. However, as the number of network layers increases, the spatial resolution of the feature map is usually reduced, which may lose some local details and boundary information. Although high-level features have important semantic information for tasks such as pixel classification, excessive downsampling may lead to the loss of some fine-grained image structures, which in turn poses a potential risk to the overall detection effect of the model. Figure 7As shown in the figure, in order to deal with this problem, we propose a multi-scale feature fusion model. The multi-scale feature fusion module extracts multi-level features from the encoder input image. However, these different levels of features have multiple differences in spatial resolution due to downsampling. Therefore, it is necessary to upsample the low-resolution feature map to make it have a similar spatial size as the high-resolution feature map. Then the upsampled low-level features are fused with the high-level features, and finally the fused features are used in subsequent convolution operations to improve the feature extraction performance of the model.
[0069] A lightweight detection method for bridge cracks, the method comprising the following steps:
[0070] Step 1) Collect crack image sets in different environments from multiple appropriate Chinese and English academic journals to obtain more diverse and widely covered samples, including but not limited to bridge cracks, to improve generalization and robustness. Then divide the collected samples into two parts: training set and test set;
[0071] Step 2) construct a bridge crack detection network model based on efficient feature fusion and multi-layer attention; the model includes an encoder that converts the rich feature information data of the bridge crack image into a specific encoding form in order to more efficiently represent and process the rich feature information data of the bridge crack image during transmission or storage, and a decoder that reconverts the data encoded by the encoder into the original crack image data. These two parts are constructed in a serial manner. For the feature extraction stage, the model structure includes multiple convolutional layers, each of which is connected to a batch normalization layer and an activation function layer. The activation function uses the ReLU function, and the convolution kernel and weight parameters are initialized by a Gaussian random function with a variance of 0.01. In the encoder part, the model uses the Mobilenetv2 network module as the backbone network to keep the model lightweight and ensure feature extraction capabilities to a certain extent. The proposed hollow space pyramid pooling structure with nested multi-layer attention is used to accurately extract rich feature information data from the bridge crack image. The decoder part adopts the proposed multi-scale feature fusion module and the high-level and low-level feature adaptive weighted feature fusion module structure combined with Transformer. The adaptive weighted feature fusion module strategically fuses the high-level features extracted from the encoder end with the low-level features extracted by the multi-scale feature extraction module to enhance the feature extraction effect, thereby enhancing the segmentation effect of the bridge crack. Finally, the obtained image is upsampled 4 times to obtain a prediction map that matches the input image resolution.
[0072] Step 3) Train the network; first, send the training sample set to the model constructed in step 2) in batches, and the training samples pass through the encoder and decoder of the model in turn. Then make certain adjustments to the momentum parameters, weight decay, batch size, learning rate, and training epochs of the model training to achieve good training results; finally, linearly combine the three mean square errors as the loss function, and perform L2 regularization constraints on the abundance matrix, iterate the training model through back propagation, adjust the model parameters, and use the multiplication rule to fine-tune the end member matrix in each iteration until convergence;
[0073] Step 4) testing the model; sending the test sample set to the model trained in step 3) to obtain the corresponding test generation result;
[0074] Step 5) Evaluate the performance of the algorithm; ablation experiments and overall comparative experiments are performed based on the three improvements of the proposed model to obtain the bridge crack detection results. At the same time, the effectiveness of the algorithm is measured by relevant evaluation indicators such as MIoU, IoU, and F1 score. The loss curve and MIoU curve of the improved model are also recorded in the experiment to reflect the experimental effect.
[0075] Take the bridge crack images we took on some bridges in Shanghai using a high-definition digital camera as an example, crop the images to a suitable size, and take a local image of 448×448 pixels as a reference image. The specific implementation example calculation steps are as follows:
[0076] Step 1) Crop the 100 bridge crack images with a resolution of 4608*3456 to obtain crack images with a cropped image size of 448*448, and divide the processed image set into two parts: a training set and a test set according to a ratio of 8 to 2.
[0077] Step 2) Build a bridge crack detection network model based on efficient feature fusion and multi-layer attention: The construction of the network model is exactly the same as that of the above technical solution. It still adopts the encoder-decoder structure and uses Mobilenetv2 as the backbone network instead of the Resnet series to achieve a more detailed balance between accuracy and speed.
[0078] Step 3) Train the network model: The training image set divided in step 1) is sent to the network training, and the PyTorch deep learning framework is used to train and test our model. The experimental device we used includes Python 3.8, CUDA 11.1, 15vCPU AMD EPYC 7543 32-core processor and RTX 3090GPU with 24GB memory. The training is performed using the stochastic gradient descent (SGD) optimizer, the momentum size is set to 0.9, and the weight decay is 0.0001. The batch size is 16 and the training rounds are 200. The learning rate is initially set to 0.007 and adjusted using the cosine annealing method. The input image size is fixed to 448×448.
[0079] Step 4) Testing the model: sending the test sample set to the above-mentioned trained model to obtain the corresponding generation result.
[0080] Step 5) Evaluate algorithm performance; see Figure 8 , depicts the crack image detection results for multiple sets of ablation experiments. The boxes represent false detections or missed detections. It is obvious that a single model enhancement may lead to different degrees of false detections or missed detections when the model identifies crack pixels. In contrast, the proposed model shows excellent segmentation performance with very few missed or false detections. This highlights the superior ability of our proposed model in accurately detecting and identifying crack pixels.
[0081] Step 6) See Fig. 9 , some pictures outside the private dataset were collected for detection. The figure shows that the detection effect is relatively good, indicating that the proposed model has good generalization ability. Fig. 9 The network model proposed in this invention performs generalized detection on different crack images that have not participated in the training. It can be clearly seen from the figure that our model shows good segmentation detection effect for general cracks, and there are almost no missed detection and false detection images, which shows that the model we proposed has strong comprehensive detection ability and generalization ability.
[0082] Step 7) See Fig.10 , the loss curve of the improved model continues to decrease and stabilize, showing effective learning of patterns and features. Compared with the original model, the enhanced model converges to a lower loss value faster. Among them, the proposed model has the smallest loss and the convergence speed is also ahead of other models, showing efficient adaptation to the characteristics of specific crack detection tasks. Its fast weight adjustment reflects excellent efficiency and robustness, highlighting its excellent performance in crack detection. This highlights the model's ability in feature learning and weight optimization to obtain the best results.
[0083] Step 8) See Fig.11, the MIoU curve in the figure reflects the performance of the model in semantic segmentation. During the training process, the MIoU of each model will rise and stabilize, indicating that the training has converged. The peak of the proposed model is the highest, indicating the highest segmentation accuracy. All improved models exceed the original model, showing excellent crack detection capabilities. The steeper the curve (especially the curve of the proposed model), the higher the learning efficiency. The fluctuation of the proposed model is the smallest, indicating higher stability. Our model improves performance in all aspects.
[0084] Step 9) Table 1 lists the results of the ablation experiments conducted on our dataset. Overall, compared with DeepLabV3Plus, our proposed model improves by 2.8% in MIoU, 5.4% in IoU, and nearly 4.2 points in F1. This directly shows that the comprehensive performance of our proposed model in the specific detection of cracks is significantly improved. The results of improvement 1 show that the model's ability to detect and identify cracks has been greatly improved through the adaptive weighted feature fusion method. The obvious improvement in MIoU, IoU and F1 shows that this method can more effectively capture key information in the process of high-level and low-level feature fusion, thereby improving the performance of the model. At the same time, the accuracy of distinguishing crack and background pixels has also been significantly improved, further verifying the effectiveness of this fusion method. As for improvement 2, although the increase in MIoU is less than 1%, both IoU and F1 have been significantly improved, indicating that the atrous spatial pyramid pooling structure with nested multi-layer attention has significantly improved the classification performance in crack and background segmentation on private datasets. Although the improvement in crack segmentation is relatively small, the overall contribution of this module to the model performance is still large. Improvement 3 shows that the slight improvement in the main indicators may stem from the limited presence of complex features and correlations in the dataset. These complex features and correlations may require more powerful models or complex feature extraction methods to effectively capture. In this case, the multi-scale feature fusion module may find it difficult to fully utilize the information in the dataset, resulting in limited performance improvement.
[0085] Table 1. Ablation test results of the model
[0086]
[0087] Step 10) In order to verify the effectiveness of the proposed method, we selected a series of typical image segmentation models for comparative testing. At the same time, we reproduced some state-of-the-art models for comparison to show the comprehensive performance of our proposed model. These lightweight and efficient segmentation models adopt the Transformer architecture. Table 2 lists the evaluation indicators obtained from the comparative experiments. The table clearly shows that among the classic segmentation models, DeepLabV3Plus outperforms other models in crack detection and recognition, especially in IOU and F1. This shows that the model can more accurately locate crack boundaries, accurately segment crack areas, and better distinguish cracks from backgrounds. The number of parameters is not explicitly provided in the table, mainly because the DeepLabV3Plus model uses mobilenetv2 as the backbone network, which has the least parameters while showing excellent detection capabilities. Compared with DeepLabV3Plus, the improved model we proposed has improved by 5.4% in IOU, 2.8% in MIoU, and nearly 4 points in F1. In addition, we also compared our model with other more advanced models, and the table shows that our proposed model achieved superior results in terms of MIoU and reached the highest value in terms of F1. This indicates that our proposed model outperforms other commonly used models, or at least is comparable to these more advanced models.
[0088] Table 2. Evaluation metrics of different models on private datasets
[0089]
[0090] The above-mentioned embodiment only expresses one implementation mode of the present invention, and its description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the attached claims.
Claims
1. A bridge crack detection method based on multi-scale feature fusion and multi-layer attention, characterized in that: The following steps are involved: Step 1) Collect crack image sets under different environments to obtain more diverse and widely covered samples, and divide the collected samples into two parts: training set and test set; Step 2) constructing a bridge crack detection network model based on efficient feature fusion and multi-layer attention; the model includes an encoder that converts the rich feature information data of the bridge crack image into a specific encoding form in order to more effectively represent and process the rich feature information data of the bridge crack image during transmission or storage, and a decoder that converts the data encoded by the encoder back into the original crack image data, and the two parts are constructed in a serial manner; For the feature extraction stage, the model structure includes multiple convolutional layers, each of which is connected to a batch normalization layer and an activation function layer. The activation function uses the linear rectified unit ReLU function, and the convolution kernel and weight parameters are initialized using a Gaussian random function with a variance of 0.
01. In the encoder part, the model uses the Mobilenetv2 network module as the backbone network to keep the model lightweight and ensure feature extraction capabilities to a certain extent. It uses a hollow spatial pyramid pooling structure with nested multi-layer attention to accurately extract rich feature information data from bridge crack images. The decoder part adopts a multi-scale feature fusion module and a high- and low-level feature adaptive weighted feature fusion module structure combined with Bottleneck Transformer. The adaptive weighted feature fusion module strategically fuses the high-level features extracted from the encoder end with the low-level features extracted by the multi-scale feature extraction module to enhance the feature extraction effect, thereby enhancing the segmentation effect of bridge cracks. Finally, the obtained image is upsampled 4 times to obtain a prediction map that matches the input image resolution. Step 3) training the network; first, the training sample set is sent to the model constructed in step 2) in batches, and the training samples pass through the encoder and decoder of the model in turn; then, the momentum parameter, weight decay, batch size, learning rate, and training epoch of the model training are adjusted to achieve a good training effect; finally, the three mean square errors are linearly combined as the loss function, and the abundance matrix is constrained by L2 regularization, the model is iteratively trained by back propagation, and the model parameters are adjusted. In each iteration, the end member matrix is fine-tuned using the multiplication rule until convergence; Step 4) testing the model; Send the test sample set to the model trained in step 3) to obtain the corresponding test generation result; Step 5) Evaluate the performance of the algorithm; Based on the three improvements of the proposed model, the ablation experiment and the overall comparative experiment are performed to obtain the bridge crack detection results. At the same time, the effectiveness of the algorithm is measured by relevant evaluation indicators such as MIoU, IoU, and F1 score. The loss curve and MIoU curve of the improved model are also recorded in the experiment to reflect the experimental effect; Among them, the structure of the high-level and low-level feature adaptive weighted feature fusion module combined with Bottleneck Transformer in step 2) is specifically as follows: First, the high-level features output by the encoder are linearly upsampled by 4 times to match the resolution of the low-level feature map, generating the upsampled feature map F 4H , as shown in formula (1); then, the upsampled high-level features and low-level features are concatenated in the channel dimension to obtain F L-4H , as shown in formula (2); then, as shown in formulas (3) and (4), the concatenated features are input into the Bottleneck Transformer module and then through the Conv 1×1 And ReLU activation function generates spatial feature map W R ; F 4H =f Up4 (F H ) (1) W L-4H =f Conv1×1 (f Trans (F L-4H )) (3) W R =δ(W L-4H ) (4) where F L represents the low-level feature map, F H represents a high-level feature map, represents feature connection, Represents bilinear interpolation; For low-level features F L and spatial feature map W R Perform adaptive weighted feature fusion to obtain W W-Add , as shown in formula (5); this can more effectively locate the target object and use spatial details to obtain finer shape and boundary information; then, the weighted features and high-level features are directly merged; the resulting feature map is then subjected to a convolution operation to obtain a refined and concentrated fusion feature as shown in formula (6); finally, this output is upsampled to match the resolution of the input image to generate a prediction map; W W-Add =Σ(W R ,F L )=ω1W R +ω2F L (5) ω1+ω2=1 (6) where f Conv1×1 Represents Conv 1×1 Operation, f Conv3×3 Represents Conv 3×3 Operation, f Trans represents the Bottleneck Transformer module, δ represents the ReLU activation function, Σ represents the adaptive weighted feature fusion operation, and ω1, ω2 represent the weighting coefficients, and their relationship is shown in Equation (6); Finally, the weighted features and high-level features are directly concatenated; the concatenated feature map undergoes a 3×3 convolution operation to generate a refined and concentrated fusion feature, as shown in formula (7); then, this output is upsampled to match the resolution of the input image to generate a prediction map; Among them, the hollow space pyramid pooling structure with nested multi-layer attention is as follows: The structure introduces channel prior convolutional attention CPCA, which effectively integrates global context by connecting multiple layers of dilated convolution and pooling structures in series to form parallel branches; secondly, CPCA dynamically allocates attention weights in channels and space to reduce redundant information. This combination improves the model's understanding of the semantic structure of the image and enables the model to focus more accurately on key semantic areas and enhance edge detail processing; then, the strategy of CPCA dynamically allocating attention weights in channels and space simplifies operations and avoids computational overhead; the structure further introduces a coordinate attention mechanism to integrate information from parallel branches and enhance the model's understanding of target location and context; ultimately, these innovations aim to improve the accuracy of image segmentation by emphasizing relevant areas and suppressing irrelevant information; Among them, the multi-scale feature fusion module is as follows: The multi-scale feature fusion module extracts multi-level features from the encoder input image, but these different levels of features have exponential differences in spatial resolution due to downsampling. Therefore, it is necessary to upsample the low-resolution feature map to make it have a similar spatial size as the high-resolution feature map; then the upsampled low-level features are fused with the high-level features, and finally the fused features are used for subsequent convolution operations to improve the feature extraction performance of the model.
2. The bridge crack detection method based on multi-scale feature fusion and multi-layer attention according to claim 1 is characterized in that: In step 1), the high-resolution bridge crack images are cropped to obtain crack images with a cropped image size of 448*448, and the processed image set is divided into two parts, a training set and a test set, according to a ratio of 8 to 2.
3. The bridge crack detection method based on multi-scale feature fusion and multi-layer attention according to claim 1 is characterized in that: In step 3), the training image set divided in step 1) is sent to the network training, and the PyTorch deep learning framework is used to train and test the model; the experimental equipment used includes Python3.8, CUDA 11.1, 15vCPU AMDEPYC 7543 32-core processor and RTX 3090GPU with 24GB memory. The stochastic gradient descent optimizer is used for training, the momentum size is set to 0.9, and the weight decay is 0.0001; the batch size is 16, and the number of training rounds is 200; the learning rate is initially set to 0.007 and adjusted using the cosine annealing method; the input image size is fixed to 448×448.
4. The bridge crack detection method based on multi-scale feature fusion and multi-layer attention according to claim 1 is characterized in that: The Bottleneck Transformer in step 2) is very lightweight in the Transformer field and can avoid long training time. It contains three tensor compressions, two layers of multi-head self-attention, and two tensor expansions. Tensor compression downsamples the image to effectively extract key features. Multi-head self-attention captures abstract and low-level features from images; Tensor expansion further processes and aggregates information in feature maps through convolution operations, and upsamples to restore image details. In addition, the model uses jump connections, which not only optimizes the efficiency of image processing, but also reduces model parameters, thereby achieving efficient and accurate image processing.
Citation Information
Patent Citations
Bridge crack detection method based on improved PSPNet network
CN112560895A
Bridge concrete crack detection method under complex background based on deep learning
CN116823800A
Oral cavity CBCT image tooth and soft tissue segmentation model method based on improved U-Net model
CN117115132A
Medical image segmentation method based on u-net
US20220309674A1
Cited By
Rock microcrack identification and detection method based on deep learning
CN119693794A
A rock micro-crack identification and detection method based on deep learning
CN119693794B
Hydraulic structure crack detection method and device, electronic equipment and storage medium
CN119722677A
Hydraulic structure crack detection method, device, electronic equipment and storage medium
CN119722677B
Lightweight concrete structure crack automatic detection method
CN119888434A
