Bridge appearance tiny damage identification method based on multi-scale feature single-sequence fusion algorithm

Abstract

The invention discloses a bridge appearance tiny damage identification method based on a multi-scale feature single-sequence fusion algorithm, and the method comprises the steps: constructing a bridge appearance tiny damage detection model, which comprises multi-scale feature fusion models CITFPN, RPN and RoI Align, and a network head architecture for outputting a prediction result; the CITFPN comprises a feature extraction network in which an attention module is introduced, and a single-sequence feature fusion module IFPN in which transposition convolution is introduced; the IFPN is used for carrying out one-time single-sequence superposition fusion on a high-semantic feature map generated in the last convolution stage of the feature extraction network and feature maps generated in other convolution stages except the first convolution stage and output by the feature extraction network, and then carrying out further convolution processing on the fused feature map; and meanwhile, down-sampling is carried out on the high-semantic feature map, and the high-semantic feature map and other feature maps after further convolution processing are fused again to serve as output of the CITFPN. According to the invention, automatic and refined segmentation of the bridge disease instance can be realized.

Description

Technical Field

[0001] The present invention relates to the field of bridge structure surface information detection, and in particular to a bridge surface subtle damage recognition method based on a multi-scale feature single sequence fusion algorithm. Background Art

[0002] In recent years, many scholars at home and abroad have carried out research on the automatic identification method of apparent damage based on deep learning, especially in the detection of various defects such as cracks, spalling and holes. However, the existing bridge inspection technology still has many problems to be solved when it comes to the identification and quantification of bridge defects in actual complex scenarios:

[0003] (1) In actual engineering applications, due to the particularity of bridge structures and the safety requirements of shooting equipment, the number of pixels occupied by the defect targets in drone images is often small, and there is a significant imbalance between the number of pixels occupied by the defect targets and the background. In addition, most images have complex backgrounds and contain a lot of noise. At present, the instance segmentation models for bridge defects are mostly trained based on crack datasets collected in laboratory environments or magnified defect images. These datasets have a large proportion of defect pixels and good image quality, which is significantly different from the actual situation. As a result, the existing models are difficult to directly apply to small target recognition tasks in actual engineering, and their application is limited.

[0004] (2) The accuracy of identifying apparent subtle damage needs to be improved.

[0005] (3) Existing identification methods involve a single type of disease, most of which focus on typical diseases such as cracks, and rarely identify multiple diseases simultaneously. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention proposes a bridge apparent subtle damage identification method based on a multi-scale feature single sequence fusion algorithm. The specific technical solution is as follows:

[0007] A bridge apparent subtle damage identification method based on a multi-scale feature single sequence fusion algorithm, the method comprising the following steps:

[0008] S1: Establish a bridge surface subtle damage dataset;

[0009] S2: Construct and train a bridge surface subtle damage detection model; the bridge surface subtle damage detection model includes a multi-scale feature fusion model CITFPN, RPN, RoI Align and a network head architecture that outputs prediction results; the multi-scale feature fusion model includes a feature extraction network that introduces an attention module, and a single-sequence feature fusion module IFPN that introduces a transposed convolution;

[0010] The multi-scale feature fusion module is used to perform a one-time single-sequence superposition fusion of the high semantic feature map generated by the last convolution stage of the feature extraction network and the feature maps generated by the remaining convolution stages except the first convolution stage output by the feature extraction network, and then further convolution the fused fusion feature map; at the same time, the high semantic feature map is downsampled; then, all the feature maps after the further convolution processing and the downsampled high semantic feature map are fused again as the output of the multi-scale feature fusion module; the output of the multi-scale feature fusion module is used as the input of the RPN, and a fixed-size feature region is generated through the RPN and RoI Align, and the corresponding mask, bounding box and category label are output through the network head architecture that outputs the prediction result;

[0011] S3: Use the trained bridge surface subtle damage detection model to automatically identify bridge surface subtle damage.

[0012] Furthermore, the feature extraction network is Resnet101, which includes five convolution stages C1, C2, C3, C4, and C5. The single-sequence feature fusion module IFPN upsamples the feature map p5 generated in the C5 stage, and performs a one-time single-sequence superposition fusion with the feature maps generated by C2, C3, and C4 to obtain p2, p3, and p4, and then performs 3×3 convolution processing on p2, p3, p4, and p5 to obtain P2, P3, P4, and P5; finally, P5 is downsampled to obtain P6, and P2, P3, P4, P5, and P6 are fused again as the output of the multi-scale feature fusion module.

[0013] Furthermore, an attention module is built into the identity block and convolution block in the feature extraction module Resnet101. The attention module includes a channel attention submodule and a spatial attention submodule, which compress and weight features in the channel dimension and spatial dimension respectively, so as to enhance detail features and suppress noise.

[0014] Furthermore, the specific steps of the attention module to compress and weight features in the channel dimension and the spatial dimension are as follows:

[0015] (1) After the input feature F1 is compressed by the average pooling layer and the maximum pooling layer, it passes through two layers of perceptrons to obtain two independent channel features. The two channel features are further added element by element, and the channel attention weight Mc is obtained through a convolutional layer with a Sigmoid activation function;

[0016] (2) multiplying the channel attention weight Mc by the input feature F1 to obtain the channel attention feature F2, inputting the channel attention feature F2 into the average pooling layer and the maximum pooling layer connected in series, and then passing through the convolution layer with a Sigmoid activation function to obtain the spatial attention weight Ms;

[0017] (3) Multiply the spatial attention weight Ms by the channel attention feature F2 to obtain the final attention enhancement feature F3.

[0018] Furthermore, the single-sequence feature fusion module IFPN upsamples the feature map p5 generated in the C5 stage using transposed convolution.

[0019] Furthermore, when training the bridge apparent subtle damage detection model, the transfer learning method is first used to transfer the training weights of the COCO dataset to initialize the model; then, a phased training method is adopted, that is, first, while freezing the other layers, the parameters of the model head layer are adjusted through training, and then, while unfreezing the other layers, all layers are trained.

[0020] A bridge apparent subtle damage recognition system based on a multi-scale feature single sequence fusion algorithm comprises one or more processors for implementing a bridge apparent subtle damage recognition method based on a multi-scale feature single sequence fusion algorithm.

[0021] An electronic device, comprising:

[0022] one or more processors;

[0023] A storage device is used to store one or more programs. When the one or more programs are executed by the electronic device, the electronic device implements a method for identifying apparent subtle damage to a bridge based on a multi-scale feature single-sequence fusion algorithm.

[0024] A computer-readable storage medium stores a program, which, when executed by a processor, implements a bridge surface subtle damage identification method based on a multi-scale feature single-sequence fusion algorithm.

[0025] The present invention has the following beneficial effects:

[0026] (1) The bridge apparent subtle damage identification method based on the multi-scale feature single-sequence fusion algorithm of the present invention adopts a single-time fusion feature strategy to reduce the network complexity while enhancing its ability to process detailed information of the disease, thereby improving the continuity and accuracy of pixel-level identification of bridge diseases, and is more suitable for the identification of minor defects in field bridge inspections.

[0027] (2) In the bridge apparent subtle damage detection model of the present invention, a parameter-learnable transposed convolution is introduced into the multi-scale feature fusion model as an upsampling method for expanding the feature map size, so that the upsampling process is continuously optimized, the texture and edge of the image are effectively reconstructed, and a clearer and more continuous image is obtained, thereby providing better accuracy and continuity for the bridge disease segmentation task.

[0028] (3) In order to reduce the impact of background noise on the accuracy of small target recognition, the multi-scale feature fusion model introduces a multiple attention mechanism to suppress useless information such as cluttered background and noise, enhance the local feature processing capability, and select a strategy with higher accuracy and lower model complexity in the selection of connection strategy. It has better performance in the segmentation of apparent subtle damage of bridges in complex environments.

[0029] (4) The bridge apparent subtle damage identification method based on the multi-scale feature single-sequence fusion algorithm of the present invention can perform image segmentation on images of larger scale, unbalanced categories, and larger background noise taken during on-site bridge inspections. It has high robustness and overcomes the problems of larger defect size and smaller background noise in traditional data set images. It is convenient to carry out high-precision and high-efficiency identification using equipment such as drones.

[0030] (5) The bridge apparent subtle damage identification method based on the multi-scale feature single-sequence fusion algorithm of the present invention realizes multi-category pixel-level identification of bridge apparent subtle damage for small targets and complex backgrounds in field bridge inspections, effectively meets the subtle damage segmentation requirements of small targets under complex backgrounds, obtains the contour, position, area and other information of subtle damage, contributes to the automated and refined detection of bridge diseases, and provides a more reliable, objective and comprehensive basis for bridge health monitoring and management. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 The present invention is a flowchart of a method for identifying apparent subtle damage to a bridge based on a multi-scale feature single-sequence fusion algorithm according to an embodiment of the present invention.

[0032] Figure 2 This is the network structure diagram of CITNET, a bridge surface subtle damage detection model.

[0033] Figure 3 This is the MSDefect dataset provided by the embodiment of the present invention.

[0034] Figure 4 An example diagram of typical defects identified by an embodiment of the present invention.

[0035] Figure 5 Comparison chart of recognition results for different CBAM connection strategies. DETAILED DESCRIPTION

[0036] The present invention will be described in detail below based on the accompanying drawings and preferred embodiments, and the purpose and effects of the present invention will become more clear. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0037] Technical term explanation

[0038] ResNet, Residual Neural Network, residual network;

[0039] FPN, Feature Pyramid Network, feature pyramid network;

[0040] CBAM, Convolutional Block Attention Module, convolutional block attention module;

[0041] RPN, Region Proposal Network, region proposal network;

[0042] RoI Align, Region of Interest Align;

[0043] CA, Channel Attention, channel attention;

[0044] SA, Spatial Attention, spatial attention;

[0045] MLP, Multi-Layer Perceptron, multi-layer perceptron;

[0046] Identity Block, identity block;

[0047] Conv Block, Convolutional Block, convolution block;

[0048] FCN, Fully Convolutional Network, full convolutional network;

[0049] Dense Layers, Dense Layers, fully connected layers;

[0050] Adam, Adaptive Moment Estimation, adaptive moment estimation;

[0051] SGD, Stochastic Gradient Descent, stochastic gradient descent;

[0052] IoU, the Intersection over Union, intersection over union ratio;

[0053] P, Precision, accuracy;

[0054] R, Recall, recall rate;

[0055] F1 Score, F1 Score, F1 score;

[0056] mAP, mean Average Precision, average precision;

[0057] like Figure 1 As shown, the bridge apparent subtle damage identification method based on the multi-scale feature single sequence fusion algorithm of this embodiment includes the following steps:

[0058] S1: Establish a bridge apparent subtle damage dataset.

[0059] The images of bridge surface subtle damage are collected by using drone-mounted cameras and other shooting equipment, and the MSDefect dataset is established based on the photos taken during on-site bridge inspection as raw data. There is at least one subtle disease in each image, and the entire bridge surface subtle damage dataset should contain subtle diseases of various forms, such as cracks, exposed steel bars and other slender tubular diseases and other conventional diseases such as pockmarks and holes, and each subtle disease category has a certain number of images to meet the needs of model training, for example, no less than 100 images.

[0060] Then, data enhancement is performed on the images in the MSDefect dataset, including angle rotation, translation, shear transformation, scaling, and horizontal flipping of the images. The dataset is divided into non-overlapping training datasets and validation datasets. Each set of data uses the original image as input data and the annotated mask image as the prediction comparison value for testing and training network performance.

[0061] S2: Build and train a bridge surface subtle damage detection model.

[0062] like Figure 2As shown in the figure, the bridge apparent subtle damage detection model CITNet includes a multi-scale feature fusion model CITFPN, RPN, RoI Align and a network head architecture that outputs prediction results; the multi-scale feature fusion model includes a feature extraction network that introduces an attention module, and a single-sequence feature fusion module IFPN that introduces transposed convolution. The complex background and class imbalance problems in bridge disease recognition are effectively alleviated through multiple attention mechanisms and feature fusion. In the single-sequence feature fusion module, a learnable transposed convolution is introduced as an upsampling method to expand the size of the feature map, so that the upsampling process is continuously optimized to obtain clearer and more continuous images, thereby providing better accuracy and continuity for the bridge disease segmentation task.

[0063] (1) Feature extraction network introducing attention module

[0064] Taking the feature extraction network Resnet101 as an example, it contains five convolution stages C1, C2, C3, C4, and C5. In the process of disease identification and segmentation, the surface image data of the bridge is first input into the feature extraction network. Resnet is a residual network structure that optimizes image processing performance through a deep network architecture while ensuring that the performance is at least equal to that of a shallow network. In order to solve the problem of performance degradation of deep networks, a shortcut connection of identity mapping between blocks is established. Specifically, each block accepts an input feature x, and the feature objective function H(x)=F(x)+x that originally needs to be learned by the layer. F(x) represents the residual part that the network block needs to learn, and x is the input feature directly added to the output through a shortcut connection. Then:

[0065] Output of layer l: x l+1 =x l +F(x l ,w l ), x l Input for layer l

[0066] Output of layer l+1: x l+2 =x l+1 +F(x l+1 ,w l+1 )=x l +F(x l ,w l )+F(x l+1 ,w l+1 )

[0067] Output of layer L:

[0068] Use the chain rule to calculate the gradient back propagation:

[0069]

[0070] The gradient of each layer is not only transmitted through the residual function F(x), but also directly transmitted through the shortcut connection of the identity mapping. This structural design ensures that the overall gradient will not disappear in the deep network, avoiding the problem of shallow parameter gradient loss caused by the chain rule. At the same time, when adding network layers fails to significantly improve performance, the residual F(x) tends to 0. At this time, the output feature y is the same as the input feature x, which is equivalent to not adding a new layer to the network, thereby ensuring that the network performance will not degrade.

[0071] The feature maps at all levels generated in the residual network resnet101 often contain rich or even redundant information. Although they can ensure a comprehensive understanding of the input data, they may also generate too many parameters, which may lead to feature information redundancy and resource waste. In order to strengthen the detail features and suppress background noise while ensuring attention to key features, the present invention introduces a CBAM module in resnet101 in the CITFPN module to improve the sensitivity to specific features. CBAM consists of a channel attention module (CA) and a spatial attention module (SA), which compress and weight features in the channel dimension and spatial dimension respectively. After the input feature F1 is compressed by the average pooling layer and the maximum pooling layer, two independent channel feature representations are obtained through a two-layer perceptron (MLP), and the two channel features are further element-wise added (element-wise addition), and the channel attention weight Mc is obtained through a convolutional layer with a Sigmoid activation function. Mc helps the model evaluate the importance of different channels, thereby emphasizing the channels that are helpful to the current task and suppressing irrelevant channels. The weight Mc is multiplied by the initial input feature F1 to obtain the channel attention feature F2. The channel attention feature F2 is input into the average pooling layer and the maximum pooling layer in series, and then passes through the convolution layer with the Sigmoid activation function to obtain the spatial attention weight Ms. Ms helps to highlight important image areas and reduce the influence of unimportant areas such as background noise. The weight Ms is multiplied by the channel attention feature F2 to obtain the final attention-enhanced feature F3. In order to improve the sensitivity to important features at different stages of generating feature maps, CBAM is added to the identity block and conv block of ResNet. In the identity block, CBAM pays more attention to the output after the jump connection of the feature map, thereby emphasizing or suppressing certain features; in the convblock, CBAM will adjust the attention when the feature map is deeply transformed, so that the features learned by the model are more concentrated. Through the application of CBAM, the network allocates limited computing resources to more important features during training, enhances the sensitivity to useful features, and suppresses the sensitivity to useless features, so that the efficiency and accuracy of the model are improved.

[0072] In addition, since the newly designed feature map connection and fusion method can realize the direct fusion of deep and shallow feature information, it avoids the problem of shallow feature information not being significant in the traditional multi-level fusion method, and thus improves the recognition accuracy of small disease details. Taking C2 and C5 as examples, C2 is the second convolution stage that the image has passed through. The resulting feature map is closer to the original image, has a higher resolution, contains more detail information, and less semantic information; C5 is the fifth convolution stage that the image has passed through. Due to multiple convolution and pooling operations, the resolution is lower, the semantic information is more enriched, but the detail information may be lost. Since the size and number of channels of the feature map at each stage are different, all feature maps must undergo a 1×1,256 channel convolution to obtain the same size and channel initial feature map p i (i=2~5). The traditional FPN uses layer-by-layer fusion to perform each convolution stage C i Specifically, before the p2 feature map is fused with p3, p3 itself has integrated the high-level feature information from p4 and p5. In this process, the underlying features are gradually dominated by the high-level features, resulting in the suppression of the original detail information. To solve this problem, IFPN changes the fusion strategy based on the FPN network architecture to:

[0073] like Figure 2 As shown in the figure, the multi-level feature extraction of the input image is realized through continuous convolution and pooling in Resnet101, and a series of multi-scale feature maps with decreasing resolution and increasing semantic information are obtained. The single-sequence feature fusion module IFPN upsamples the feature map p5 generated in the C5 stage, and then performs a one-time single-sequence superposition fusion with the feature maps generated by C2, C3, and C4 to obtain p2, p3, and p4. This single-time fusion method effectively reduces the number of repeated fusions between low-level features and high-level features, thereby reducing the risk of the underlying detail information being submerged by the high-level feature information during the fusion process. By increasing the proportion of shallow features in the final fused feature map, IFPN can more accurately capture small targets and detail information in the image, which helps to improve the performance of fine-grained target detection. In addition, the one-time fusion strategy also helps to simplify the network structure and reduce the computational burden, so that IFPN can achieve higher efficiency in practical applications while maintaining the advantage of feature fusion. Then p2, p3, p4, and p5 are processed by 3×3 convolution to obtain P2, P3, P4, and P5 to reduce the aliasing effect of upsampling. Finally, after downsampling P5, a reference feature map P6 is generated, which is more conducive to generating prediction results. P2, P3, P4, P5, and P6 are fused again as the output of the multi-scale feature fusion module to obtain the final feature map used for RPN to perform target classification and boundary regression prediction.

[0074] Finally, in order to enhance detail restoration and image reconstruction in the image segmentation process, the single-sequence feature fusion module IFPN of the present invention introduces transposed convolution as an upsampling method when upsampling the feature map p5 generated in the C5 stage. The upsampling weights are adaptively optimized through the training process, and the texture and edges of the image are effectively reconstructed to obtain clearer and continuous images, generate richer feature representations, and avoid edge blurring that may be caused by conventional upsampling, thereby providing better accuracy and continuity for the bridge disease segmentation task.

[0075] The extracted feature map is further input into RPN and RoI Align to generate a fixed-size feature region, and the fully convolutional layer and fully connected layer are used to output the corresponding mask, bounding box and category label.

[0076] When training the bridge surface subtle damage detection model CITNet, a reasonable optimization method is used to accelerate model convergence. Adam is used as the training optimizer, and stochastic gradient descent (SGD) is used for model training.

[0077] The model has three prediction tasks, and its loss function can be expressed as:

[0078] L=L cls +L box +L mask

[0079] Among them, L cls Represents the classification loss value, L box Represents the regression loss value of the bounding box; L mask Indicates the loss value of the masked part.

[0080] In order to improve the training efficiency of the model, when training the bridge surface subtle damage detection model, the transfer learning method is first used to transfer the training weights of the COCO dataset to initialize the model; then, a phased training method is adopted, that is, first adjusting the parameters of the model head layer through training while freezing other layers, and then training all layers while unfreezing other layers.

[0081] S3: Use the trained bridge surface subtle damage detection model to automatically identify bridge surface subtle damage.

[0082] Corresponding to the aforementioned embodiment of the method for identifying apparent subtle damages of a bridge based on a multi-scale feature single-sequence fusion algorithm, the present invention also provides an embodiment of an apparatus for identifying apparent subtle damages of a bridge based on a multi-scale feature single-sequence fusion algorithm.

[0083] An embodiment of the present invention provides a device for identifying apparent subtle damage to a bridge based on a multi-scale feature single-sequence fusion algorithm, comprising one or more processors for implementing the method for identifying apparent subtle damage to a bridge based on a multi-scale feature single-sequence fusion algorithm in the above embodiment.

[0084] The embodiment of the bridge apparent subtle damage identification device based on the multi-scale feature single sequence fusion algorithm of the present invention can be applied to any device with data processing capability, and the any device with data processing capability can be a device or apparatus such as a computer. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capability in which it is located to read the corresponding computer program instructions in the non-volatile memory into the memory and run it. From the hardware level, the bridge apparent subtle damage identification device based on the multi-scale feature single sequence fusion algorithm of the present invention is a hardware structure of any device with data processing capability. In addition to the processor, memory, network interface, and non-volatile memory, any device with data processing capability in which the device in the embodiment is located can also include other hardware according to the actual function of the device with data processing capability, which will not be described in detail.

[0085] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0086] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0087] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the method for identifying apparent subtle damage to a bridge based on a multi-scale feature single-sequence fusion algorithm in the above embodiment is implemented.

[0088] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of any device with data processing capability and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0089] Example

[0090] In this example, a total of 2275 images of bridge surface subtle damage were selected, including 294 exposed reinforcement images, 1500 cracks, 375 holes, and 106 rough surfaces. Considering the hardware resource cost and computational complexity of neural network data processing, we appropriately cropped the sample images into images with a resolution of 800*600 and annotated them using Labelme software. Figure 3 The MSDefect dataset constructed in the embodiment is shown. The dataset is divided into non-overlapping training dataset and validation dataset, and each set of data uses the original image as input data and the annotated mask image as a prediction comparison value for testing and training network performance.

[0091] The bridge surface subtle defect images in the data set are used as the input data for model training. The ReLU activation parameter is set to 0.2, and the number of iterations is 200-500. If the index requirements are not met, the hyperparameter optimization is continued. If the index is met, the training weights are saved to obtain the bridge surface subtle damage recognition model CITNet after training. The hyperparameter values ​​set in this example are detailed in Table 1.

[0092] Table 1 CITNet model hyperparameter settings

[0093]

[0094] In this embodiment, IOU, precision P, recall rate R, F1 score and mean average precision mAP are used to evaluate the recognition performance of the bridge surface subtle damage detection model.

[0095] IOU (the Intersection over Union) represents the ratio of the overlapping area to the total area of ​​the actual area and the predicted area, and is used to evaluate the accuracy of instance segmentation. The higher the IoU value, the better the instance segmentation performance. The formula is:

[0096]

[0097] Among them, S overlap is the overlapping area between the actual area and the predicted area, S union is the total area of ​​the actual area and the predicted area.

[0098] Precision P refers to the proportion of true positive samples among all samples predicted to be positive, which is used to evaluate the model's ability to distinguish negative samples. Its calculation formula is:

[0099]

[0100] Among them, the positive samples predicted by the model as positive are TP (true positive); the negative samples predicted by the model as negative are TN (true negative); the negative samples predicted by the model as positive are FP (false positive); and the positive samples predicted by the model as negative are FN (false negative). Compared with classification tasks, instance segmentation has higher requirements, so the standard for correct identification is also higher. The detection is correct only when the defect type of the detection area is consistent with the ground truth area and the IOU between the two areas reaches at least a certain value.

[0101] The recall rate R is the proportion of actual positive samples that are correctly identified as positive samples, which is used to evaluate the model's ability to identify positive samples. Its calculation formula is:

[0102]

[0103] F1score is the harmonic mean of precision and recall, and is a comprehensive indicator for measuring the accuracy of model segmentation:

[0104]

[0105] The mean average precision (mAP) is a comprehensive measure of the overall accuracy of the model. After obtaining the values ​​of precision and recall, a precision-recall (PR) curve is plotted with precision as the vertical axis and recall as the horizontal axis. The area enclosed by the PR curve and the coordinate axis represents the AP (mean average precision) value of a specific defect type, expressed as

[0106]

[0107] mAP is the average of all class AP values, expressed as

[0108]

[0109] Where n represents the number of categories, AP i Represents the AP value of the i-th category. The bounding box mAP calculated with an IOU threshold of 0.5 is denoted as mAP bb, and the mAP of the mask calculated with an IOU threshold of 0.5 is expressed as mAP m .

[0110] like Figure 4 As shown in the figure, CITNet can automatically identify various defect images well. The recognition accuracy of each defect has been improved to a certain extent. The mAP of pitting surface has increased from 20.75% to 45.28%, an increase of 118.18%; the mAP of crack has increased from 5.26% to 28.49%, an increase of 441.32%; the mAP of hole has increased from 0 to 77.81%; the mAP of exposed reinforcement has increased from 0 to 42.31%. Through the model comparison experiment (see Table 2 for the configuration of the comparison experiment), it can be found that, taking pitting surface and hole as examples, each module in CITNet has a positive impact on the model. When all modules are integrated, the best experimental results can be achieved. For pitting surface, all four models successfully detected the target, but the segmentation results of the baseline model had most of the wrong recognition. For C and CT, most of the defect locations can be detected, but the segmented masks are smaller than the ground truth or incorrect. CFT segmented more correct results. For holes, the baseline model failed to detect defects. After inserting the CBAM block, IFPN block, and transposed convolution, more defects were successfully detected, and the mAP values ​​also increased to varying degrees. Like the pockmarked surface, the CBAM block, IFPN block, and transposed convolution all had a positive impact on the segmentation task, and CFT performed the best among the four models. It can be found that CITNet can also perform well when the background is complex and accounts for a small proportion.

[0111] Table 2 Comparative experimental settings

[0112]

[0113] At the same time, for different CBAM connection methods, Figure 5 The model conditions under different CBAM connection modes are shown in the figure. The connection ratio ( Figure 5 a) and parameters ( Figure 5 b) shows the complexity of the model, mAP( Figure 5 c) and mIOU( Figure 5 e) demonstrates the accuracy of the model, Figure 5d shows the specific connection of different connection schemes, 2-5 refers to stage2-stage5, I refers to identity block, and C refers to conv block. Through experiments, we can find that some sparsely connected networks can surpass the accuracy of the full-add CBAM network, such as some cases where the connection ratio is between 0.03 and 0.09. At the same time, some networks perform worse than the original network without CBAM, which indicates that some connection schemes may impair model performance. In addition, even if the connection ratio is the same, the accuracy and network complexity may be different, some of which will improve network performance while others will impair network performance. Therefore, choosing a connection scheme with smaller connections and higher accuracy can not only reduce network complexity, but also avoid the negative impact of the attention mechanism module in some cases, and minimize the computational cost while ensuring the prediction effect.

[0114] Those skilled in the art can understand that the above are only preferred examples of the invention and are not intended to limit the invention. Although the invention is described in detail with reference to the above examples, those skilled in the art can still modify the technical solutions recorded in the above examples or replace some of the technical features therein with equivalents. Any modification, equivalent replacement, etc. made within the spirit and principle of the invention shall be included in the protection scope of the invention.

Claims

1. A bridge apparent subtle damage identification method based on multi-scale feature single sequence fusion algorithm, characterized in that: The method comprises the following steps: S1: Establish a data set of bridge apparent subtle damage; S2: Construct and train a bridge surface subtle damage detection model; the bridge surface subtle damage detection model includes a multi-scale feature fusion model CITFPN, RPN, RoIAlign and a network head architecture that outputs prediction results; the multi-scale feature fusion model includes a feature extraction network that introduces an attention module, and a single-sequence feature fusion module IFPN that introduces a transposed convolution; The multi-scale feature fusion module is used to perform a one-time single-sequence superposition fusion of the high semantic feature map generated by the last convolution stage of the feature extraction network and the feature maps generated by the remaining convolution stages except the first convolution stage output by the feature extraction network, and then further convolution the fused fusion feature map; at the same time, the high semantic feature map is downsampled; then, all the feature maps after the further convolution processing and the downsampled high semantic feature map are fused again as the output of the multi-scale feature fusion module; the output of the multi-scale feature fusion module is used as the input of the RPN, and a fixed-size feature region is generated through the RPN and RoI Align, and the corresponding mask, bounding box and category label are output through the network head architecture that outputs the prediction result; S3: Use the trained bridge surface subtle damage detection model to automatically identify bridge surface subtle damage.

2. The bridge apparent subtle damage identification method based on multi-scale feature single sequence fusion algorithm according to claim 1 is characterized in that: The feature extraction network is Resnet101, which includes five convolution stages C1, C2, C3, C4, and C5. The single-sequence feature fusion module IFPN upsamples the feature map p5 generated in the C5 stage, and then performs a one-time single-sequence superposition fusion with the feature maps generated by C2, C3, and C4 to obtain p2, p3, and p4, and then performs 3×3 convolution processing on p2, p3, p4, and p5 to obtain P2, P3, P4, and P5; finally, P5 is downsampled to obtain P6, and P2, P3, P4, P5, and P6 are fused again as the output of the multi-scale feature fusion module.

3. The bridge apparent subtle damage identification method based on multi-scale feature single sequence fusion algorithm according to claim 1 is characterized in that: The identity block and convolution block in the feature extraction module Resnet101 have built-in attention modules, which include a channel attention submodule and a spatial attention submodule, which compress and weight features in the channel dimension and the spatial dimension respectively, so as to enhance detail features and suppress noise.

4. The bridge apparent subtle damage identification method based on multi-scale feature single sequence fusion algorithm according to claim 3 is characterized in that: The specific steps of the attention module to compress and weight features in the channel dimension and spatial dimension are as follows: (1) After the input feature F1 is compressed by the average pooling layer and the maximum pooling layer, it passes through two layers of perceptrons to obtain two independent channel features. The two channel features are further added element by element, and the channel attention weight Mc is obtained through a convolutional layer with a Sigmoid activation function; (2) multiplying the channel attention weight Mc by the input feature F1 to obtain the channel attention feature F2, inputting the channel attention feature F2 into the average pooling layer and the maximum pooling layer connected in series, and then passing through the convolution layer with a Sigmoid activation function to obtain the spatial attention weight Ms; (3) Multiply the spatial attention weight Ms by the channel attention feature F2 to obtain the final attention enhancement feature F3.

5. The bridge apparent subtle damage identification method based on multi-scale feature single sequence fusion algorithm according to claim 2 is characterized in that: The single-sequence feature fusion module IFPN upsamples the feature map p5 generated in the C5 stage using transposed convolution.

6. The bridge apparent subtle damage identification method based on multi-scale feature single sequence fusion algorithm according to claim 1 is characterized in that: When training the bridge apparent subtle damage detection model, the transfer learning method is first used to transfer the training weights of the COCO dataset to initialize the model; then, a phased training method is adopted, that is, first, the parameters of the model head layer are adjusted through training while freezing other layers, and then all layers are trained while unfreezing other layers.

7. A bridge apparent subtle damage identification system based on multi-scale feature single sequence fusion algorithm, characterized in that: The method comprises one or more processors for implementing the bridge apparent subtle damage identification method based on a multi-scale feature single sequence fusion algorithm as described in any one of claims 1 to 6.

8. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs. When the one or more programs are executed by the electronic device, the electronic device implements the bridge apparent subtle damage identification method based on the multi-scale feature single-sequence fusion algorithm as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, the method for identifying bridge apparent subtle damage based on a multi-scale feature single-sequence fusion algorithm as described in any one of claims 1 to 6 is implemented.

Images