Bridge apparent slender disease identification method based on parallel convolution adaptive segmentation model

Abstract

The invention discloses a bridge apparent slender disease identification method based on a parallel convolution adaptive segmentation model, and the method comprises the steps: a constructed bridge apparent slender disease detection model comprises a feature extraction network with a built-in parallel feature extraction module DCM, adaptive spatial feature fusion models ASFF, RPN, RoI Align and a network head architecture; dCM is a parallel module which is embedded in each stage of a feature extraction network and fuses standard convolution and dynamic snakelike convolution, each stage is interspersed in a parallel mode, each stage dynamically adjusts a convolution kernel through iteration offset by using the dynamic snakelike convolution, the morphological features of the bridge apparent slender disease are further sensed, and the morphological features of the bridge apparent slender disease are extracted. Carrying out global information constraint on the other branch by utilizing standard convolution; and the ASFF performs layer-by-layer superposition fusion on the high-semantic feature map of the last convolution stage and other feature maps except the first convolution stage, and realizes feature adaptive fusion through learnable parameters. According to the invention, automatic and refined detection of bridge appearance slender diseases can be realized.

Description

Technical Field

[0001] The present invention relates to the field of bridge structure surface information detection, and in particular to a bridge surface slenderness defect recognition method based on a parallel convolution adaptive segmentation model. Background Art

[0002] In recent years, many scholars at home and abroad have conducted extensive research on the automatic identification of apparent defects based on deep learning, especially in the detection of various bridge defects such as cracks, spalling and holes. However, the existing bridge detection technology still faces many challenges in the recognition and quantification of slender bridge defects in actual complex scenes:

[0003] (1) In actual engineering applications, due to the particularity of bridge structures and the safety requirements of shooting equipment, the defect targets in drone images are often small, and there is a significant imbalance in the number of pixels occupied by the defect targets and the background. In addition, most images have complex backgrounds and a lot of noise. The current bridge defect instance segmentation models are mostly trained based on crack datasets collected in laboratory environments or magnified defect images, which are significantly different from actual scenes. As a result, existing models are difficult to directly apply to recognition tasks in actual engineering, and their application is limited.

[0004] (2) The recognition accuracy of slenderness disease needs to be further improved. Most automatic detection methods are not general enough in terms of the morphology of slenderness disease.

[0005] (3) The existing identification methods involve a single type of disease, most of which focus on typical diseases such as cracks, and rarely identify other slender diseases such as exposed steel bars. In addition, there are difficulties in identifying other conventional diseases at the same time. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention proposes a bridge apparent slenderness disease recognition method based on a parallel convolution adaptive segmentation model. The specific technical solution is as follows:

[0007] A bridge apparent slenderness disease recognition method based on a parallel convolution adaptive segmentation model, the method comprising the following steps:

[0008] S1: Construct a dataset of bridge apparent slenderness damage;

[0009] S2: Construct a bridge apparent slenderness disease detection model DCANet, which includes a feature extraction network with a built-in parallel feature extraction module DCM, an adaptive spatial feature fusion model ASFF, RPN, RoI Align, and a network head architecture that outputs prediction results; the parallel feature extraction module DCM is a parallel module that integrates standard convolution Conv and dynamic snake convolution DSConv, and the feature extraction of each stage is interspersed in parallel, wherein a single branch uses the dynamic snake convolution method to further perceive the morphological characteristics of the bridge apparent slenderness disease, and the other branch uses the standard convolution to constrain the global information; the parallel feature extraction module DCM is embedded in the identity block and convolution block of each stage of the feature extraction network;

[0010] The adaptive spatial feature fusion model ASFF performs layer-by-layer superposition and fusion of the high semantic feature map generated by the last convolution stage of the feature extraction network and the feature maps generated by the remaining convolution stages except the first convolution stage output by the feature extraction network to obtain a multi-scale feature map Pi, i=2~k; then an upsampling or downsampling operation F is performed on Pi il , and according to the characteristics of the feature maps at each level, the spatial weight vectors at each level are obtained Finally, adaptively fuse the feature maps at all levels to obtain the feature maps P′i at all levels; then perform convolution processing on P′i to obtain P″i; downsample P″k to obtain P″(k+1), and fuse P″2~P″(k+1) again as the output of the adaptive spatial feature fusion model ASFF;

[0011] The dynamic snake convolution DSConv dynamically adjusts the convolution kernel through iterative offset so that it can better cover the shape of the disease and extract features from the disease image area.

[0012] S3: Use the trained bridge apparent slenderness defect detection model DCANet to automatically identify bridge apparent slenderness defects.

[0013] Furthermore, the execution steps of the parallel feature extraction module DCM are as follows:

[0014] (1) Initializing the parallel convolution module in the parallel feature extraction module DCM;

[0015] (2) Feature extraction is performed on the input disease feature map through standard convolution and dynamic snake convolution, where the parameters of the dynamic snake convolution need to be continuously adjusted according to the image features and model performance;

[0016] (3) The feature maps generated by the two convolutions are integrated, and the output weights of the two convolutions are adjusted and optimized according to the model performance and image features to ensure effective feature fusion and enhance the model's ability to recognize slender and complex diseases.

[0017] Furthermore, the execution steps of the dynamic snake convolution DSConv are as follows:

[0018] (1) Determine the center coordinate starting point K i±c =(x i±c ,y i±c ), c = {0, 1, 2, 3, 4}, c is the horizontal distance from the center grid, and the center coordinate starting point is used as the starting point of the convolution kernel iterative offset;

[0019] (2) Select the next position of each target to be processed for observation in turn, thereby ensuring the continuity of attention and not spreading the perception range too far due to large deformation offsets. i Initially, each grid K i±c The position depends on the previous grid K i±c-1 Position, that is, each grid K i±c In the previous grid K i±c-1 Add an offset Δ={δ|δ∈[-1,1]} to the basis, and the same applies to the y-axis direction;

[0020] (3) If the convolution kernel sensing area formed by the point selected in step (2) contains part of the location where the disease is located, then select this point as the starting point for the next convolution iteration; otherwise, return to step (2) and continue to select a new direction for offset;

[0021] (4) Smoothing the convolution kernel perception area selected in step (3) by bilinear interpolation;

[0022] (5) Repeat steps (2) to (4) so ​​that the convolution kernel is iteratively shifted to a position that can better cover the shape of the disease, and perform feature extraction of the corresponding image area.

[0023] Furthermore, the feature extraction network is Resnet101, which includes five convolution stages C1, C2, C3, C4, and C5; the specific workflow of the adaptive spatial feature fusion model ASFF is as follows:

[0024] (1) The Resnet101 network with an embedded DCM module upsamples the feature map P5 generated in the C5 stage, and then stacks and fuses it layer by layer with the feature maps generated by C2, C3, and C4 to obtain multi-scale feature maps P2, P3, and P4;

[0025] (2) For the first level feature map output Pl, the remaining three feature maps are upsampled or downsampled F il , in order to carry out subsequent adaptive feature fusion;

[0026] (3) According to the characteristics of feature maps at each level, the spatial weight vectors at each level are obtained

[0027] (4) Adaptively fuse feature maps at all levels according to the formula Obtain the characteristic graphs P'2~P'5 at each level of scale;

[0028] (5) During the training process, the weight vector is continuously optimized and updated to achieve an adaptive fusion effect.

[0029] Furthermore, the loss function of the bridge apparent slenderness disease detection model DCANet during training is a weighted sum of the classification loss value, the regression loss value of the bounding box and the loss value of the mask; the weight of the loss function of each category is different.

[0030] Furthermore, when training the bridge apparent slenderness disease detection model DCANet, the transfer learning method is first used to transfer the training weights of the COCO dataset to initialize the model; then, a phased training method is adopted, that is, first, while freezing the other layers, the parameters of the model head layer are adjusted through training, and then, while unfreezing the other layers, all layers are trained.

[0031] A bridge apparent slenderness defect recognition system based on a parallel convolution adaptive segmentation model comprises one or more processors for implementing a bridge apparent slenderness defect recognition method based on a parallel convolution adaptive segmentation model.

[0032] An electronic device, comprising:

[0033] one or more processors;

[0034] A storage device is used to store one or more programs. When the one or more programs are executed by the electronic device, the electronic device implements a method for identifying bridge apparent slenderness defects based on a parallel convolution adaptive segmentation model.

[0035] A computer-readable storage medium stores a program, which, when executed by a processor, implements a bridge apparent slenderness defect recognition method based on a parallel convolution adaptive segmentation model.

[0036] The beneficial effects of the present invention are as follows:

[0037] (1) The present invention designs a parallel feature extraction model (DCM), which realizes multi-level feature extraction of the input image through continuous parallel convolution and pooling operations fused with standard convolution and dynamic snake convolution, further perceives the slender and complex morphological characteristics of the apparent slender diseases of the bridge by using the dynamic snake convolution method, constrains the global information by using the standard convolution, and captures the global and local information of the disease image through the parallel structure. It not only improves the efficiency of feature extraction, but also better covers the shape of the disease by dynamically adjusting the shape and position of the snake convolution kernel, enhances the processing ability of the global and detailed information of the slender diseases, and improves the detection accuracy. It is more suitable for the identification needs of slight and slender diseases in field bridge inspections.

[0038] (2) A learnable and optimizable adaptive spatial feature fusion model ASFF is introduced into the bridge apparent slenderness defect detection model DCANet to perform adaptive feature fusion, so that the inconsistency problem between different feature scales is alleviated, and the conflicting information between spatial levels is alleviated to a certain extent, and the defect features that retain more detailed features are obtained, thereby improving the accuracy of defect detection.

[0039] (3) The intelligent recognition technology for apparent slender bridge defects can perform image segmentation on slender defect images with large scale, unbalanced categories, large background noise, and complex defect morphology taken during field bridge inspections. This alleviates the impact of problems such as large defect size, small background noise, and relatively simple defect morphology in traditional data set images. It facilitates the use of drones and other equipment to achieve high-precision and high-efficiency defect recognition, significantly improves the automation and refinement of bridge defect detection capabilities, and provides more reliable, objective, and comprehensive data support for bridge health monitoring and management. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 The present invention is a flow chart of the method for identifying apparent slenderness defects of bridges based on a parallel convolution adaptive segmentation model.

[0041] Figure 2 This is a network structure diagram of the bridge apparent slenderness disease detection model of the present invention.

[0042] Figure 3 Schematic diagram of the MSDefect data set according to an embodiment of the present invention.

[0043] Figure 4 This is an example diagram of typical disease identification obtained according to an embodiment of the present invention.

[0044] Figure 5 Comparison chart of recognition results for different Dsconv positions. DETAILED DESCRIPTION

[0045] The present invention will be described in detail below based on the accompanying drawings and preferred embodiments, and the purpose and effects of the present invention will become more clear. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0046] Technical term explanation

[0047] ResNet, Residual Neural Network, residual network;

[0048] FPN, Feature Pyramid Network, feature pyramid network;

[0049] RPN, Region Proposal Network, region proposal network;

[0050] RoI Align,Region of Interest Align;

[0051] Dsconv, Dynamic Snake Convolution, dynamic snake convolution;

[0052] Conv, Convolution, standard convolution;

[0053] ASFF, Adaptive Spatial Feature Fusion, adaptive spatial feature fusion;

[0054] Bilinear Interpolation, bilinear interpolation;

[0055] Baseline, baseline model;

[0056] Identity Block, identity block;

[0057] Conv Block,Convolutional Block,Convolutional Block;

[0058] FCN, Fully Convolutional Network, Fully Convolutional Network;

[0059] Dense Layers,Dense Layers,Fully connected layers;

[0060] Adam, Adaptive Moment Estimation, adaptive moment estimation;

[0061] SGD, Stochastic Gradient Descent, stochastic gradient descent;

[0062] IoU, the Intersection over Union, intersection over union ratio;

[0063] P, Precision, accuracy;

[0064] R,Recall, recall rate;

[0065] F1 Score,F1 Score,F1 score;

[0066] mAP, mean Average Precision, average precision.

[0067] like Figure 1 As shown, the bridge apparent slenderness defect recognition method based on the parallel convolution adaptive segmentation model of the present invention comprises the following steps:

[0068] S1: Construct a dataset of bridge apparent slenderness defects.

[0069] The images of bridge surface subtle damage are collected by using drone-mounted cameras and other shooting equipment, and the photos taken during on-site bridge inspection are used as raw data to establish a bridge surface slender disease dataset. The dataset has the following characteristics: (1) There is at least one slender disease in each image, especially slender diseases with small and fragile local structures and complex and changeable global morphology, such as cracks and exposed steel bars. In addition, each type of disease has a certain number of image samples in the dataset to meet the needs of model training, for example, no less than 100 images.

[0070] Then, a series of data augmentation operations are performed on the images in the bridge apparent slenderness disease dataset, including angle rotation, translation, shear transformation, scaling, and horizontal flipping, to enhance the model's adaptability to different deformations and perspectives. The dataset is then divided into non-overlapping training and validation sets, each of which uses the original image as input data, and the annotated mask image is used as a prediction comparison value to test and train the performance of the network.

[0071] S2: Construct the bridge apparent slenderness disease detection model DCANet.

[0072] Focusing on the problem of identifying slender defects in complex backgrounds faced by bridge defect detection, this paper proposes a bridge apparent slender defect detection model DCANet, such as Figure 2As shown in the figure, the parallel feature extraction model (DCM) effectively enhances the processing ability of the global and detailed information of slender defects, improves the detection accuracy, and is more suitable for the identification needs of slight and slender defects in field bridge inspections, alleviating the problem of complex and diverse defect morphologies in bridge defect identification; in the bridge apparent slender defect detection model DCANet, a learnable and optimizable adaptive spatial feature fusion model ASFF is introduced to perform adaptive feature fusion, so that the inconsistency problem between different feature scales is alleviated, and the conflict information between spatial levels is alleviated to a certain extent, which is more suitable for the identification needs of small target defects, unbalanced categories, and complex backgrounds in field bridge inspections.

[0073] In order to accurately obtain the number, area and contour shape of defects, such as Figure 2 As shown in the figure, the bridge apparent slenderness disease detection model DCANet includes a feature extraction network with a built-in parallel feature extraction module DCM, an adaptive spatial feature fusion model ASFF, RPN, RoI Align, and a network head architecture that outputs prediction results.

[0074] Taking Resnet101 as an example, it contains five convolution stages C1, C2, C3, C4, and C5. In the process of disease recognition and segmentation, the apparent image data of the bridge is first input into the feature extraction network. For the parallel feature extraction module DCM, the parallel feature extraction module DCM is a parallel module that integrates the standard convolution Conv and the dynamic snake convolution DSConv. The feature extraction of each stage is interspersed in parallel. One branch uses the dynamic snake convolution method to further perceive the morphological characteristics of the apparent slender disease of the bridge, and the other branch uses the standard convolution kernel to constrain the global information; the parallel feature extraction module DCM is embedded in the identity block and convolution block of each stage of the feature extraction network. The dynamic snake convolution Dsconv has a great advantage in flexibly focusing on the complex geometric features of the disease. The dynamic snake convolution has good dynamic structural adaptability in the x-axis and y-axis directions, and can pay more attention to the local features of the object, especially the slender structure. Define the center coordinate of the convolution kernel as K i =(x i ,y i ), the 3*3 convolution kernel with a dilation rate of 1 can be expressed as:

[0075] K={(x-1,y-1),(x-1,y),…,(x+1,y+1)

[0076] The dynamic snake convolution does not always exist as a standard 3*3 convolution kernel, but iteratively shifts. By selecting the next position of each target to be processed in turn for observation, the continuity of attention is ensured, and the perception range will not be spread too far due to large deformation shifts.

[0077] The execution steps of dynamic snake convolution DSConv are as follows:

[0078] (1) Determine the center coordinate starting point K i±c =(x i±c ,y i±c ), c = {0, 1, 2, 3, 4}, c is the horizontal distance from the center grid, and the center coordinate starting point is used as the starting point of the convolution kernel iterative offset;

[0079] (2) Select the next position of each target to be processed for observation in turn, thereby ensuring the continuity of attention and not spreading the perception range too far due to large deformation offsets. i Initially, each grid K i±c The position depends on the previous grid K i±c-1 Position, that is, each grid K i±c In the previous grid K i±c-1 Add an offset Δ={δ|δ∈[-1,1]} to the basis, and the same applies to the y-axis direction;

[0080]

[0081] (3) If the convolution kernel sensing area formed by the point selected in step (2) contains part of the location where the disease is located, then select this point as the starting point for the next convolution iteration; otherwise, return to step (2) and continue to select a new direction for offset;

[0082] (4) Since the offset may be a decimal, while the coordinates are usually in integer form, it is necessary to smooth the convolution kernel perception area selected in step (3) through bilinear interpolation;

[0083] K=∑ K′ B(K′,K)·K′

[0084] B(K,K′)=b(K x ,K x ′)·b(K y ,K y ′)

[0085] where K′ is an enumeration of all integer spatial positions, B(K′,K) is a bilinear weight function, and b(K x ,K x ′) and b(K y ,K y ′) correspond to the one-dimensional interpolation kernel functions in the x and y directions respectively.

[0086] (5) Repeat steps (2) to (4) so ​​that the convolution kernel is iteratively shifted to a position that can better cover the shape of the disease, and perform feature extraction of the corresponding image area.

[0087] Compared with standard convolution, dynamic snake convolution can better cover the overall shape of the disease by dynamically adjusting the shape and position of the convolution kernel, capturing the overall and local features of the disease, and enhancing the network's perception of the target, especially slender diseases. This improves the recognition accuracy of DCANet when segmenting slender diseases such as cracks and exposed steel bars. At the same time, due to the adaptive advantage of DSConv in identifying diseases of different sizes, DCANet has better accuracy and adaptability when segmenting the same disease with similar features but different sizes.

[0088] In the actual application of DCM, the input image (or feature map) is first extracted through standard convolution and dynamic snake convolution respectively, and the parameters of the dynamic snake convolution need to be continuously adjusted according to the image features and model performance; then the feature maps generated by the two convolution layers are integrated, and the output weights of the two are adjusted and optimized according to the model performance and image features to ensure effective feature fusion and enhance the model's ability to recognize slender and complex diseases. Through this parallel structure, the model can capture the global and local information of the disease image at the same time, which not only improves the efficiency of feature extraction, but also helps to improve detection accuracy by dynamically adjusting the shape and position of the convolution kernel to better cover the shape of the disease.

[0089] The parallel feature extraction module DCM is embedded in the identity block and convolution block of each stage of the feature extraction network to optimize feature capture and extraction. Resnet is a residual network structure that optimizes image processing performance through a deep network architecture while ensuring that the performance is at least equal to that of a shallow network. In order to solve the problem of performance degradation of deep networks, a shortcut connection of identity mapping between blocks is established. Specifically, each block accepts an input feature x, and the feature objective function H(x)=F(x)+x that originally needs to be learned by the layer. F(x) represents the residual part that the network block needs to learn, and x is the input feature directly added to the output through a shortcut connection. Then:

[0090] Output of layer l: x l+1 =x l +F(x l ,w l ), x l Input for layer l

[0091] Output of layer l+1: x l+2 =x l+1 +F(x l+1 ,w l+1 )=x l +F(x l ,w l )+F(x l+1 ,w l+1 )

[0092] Output of layer L:

[0093] Use the chain rule to calculate the gradient back propagation:

[0094]

[0095] The gradient of each layer is not only transmitted through the residual function F(x), but also directly transmitted through the shortcut connection of the identity mapping. This structural design ensures that the overall gradient will not disappear in the deep network, avoiding the problem of shallow parameter gradient loss caused by the chain rule. At the same time, when adding network layers fails to significantly improve performance, the residual F(x) tends to 0. At this time, the output feature y is the same as the input feature x, which is equivalent to not adding a new layer to the network, thereby ensuring that the network performance will not degrade.

[0096] Through continuous convolution and pooling in the feature extraction network of the built-in parallel feature extraction module DCM, multi-level feature extraction of the input image is achieved, and a series of multi-scale feature maps with decreasing resolution and increasing semantic information are obtained. Then, the adaptive spatial feature fusion model ASFF performs adaptive feature fusion on the disease features respectively to improve the network's ability to process diseases in a refined manner. Since the adaptive spatial feature fusion model ASFF has learnable features, it can suppress conflicting information and improve the invariance of feature scales by learning spatial inter-layer weights without increasing additional computational costs, thereby improving the recognition accuracy of minor slender diseases.

[0097] The adaptive spatial feature fusion model ASFF stacks and fuses the high semantic feature map generated by the last convolution stage of the feature extraction network with the feature maps generated by the remaining convolution stages of the feature extraction network output except the first convolution stage layer by layer to obtain a multi-scale feature map Pi, i = 2~k; then Pi is upsampled or downsampled F il , and according to the characteristics of the feature maps at each level, the spatial weight vectors at each level are obtained Finally, the feature maps at all levels are adaptively fused to obtain the scale feature maps P′i at all levels; P′i is then convolved to obtain P″i; P″k is downsampled to obtain P″(k+1), and P″2~P″(k+1) are fused again as the output of the adaptive spatial feature fusion model ASFF.

[0098] Taking the feature extraction network Resnet101 as an example, it contains five convolution stages C1, C2, C3, C4, and C5. The multi-scale feature maps P2 to P5 (corresponding to convolution stages C2 to C5, respectively) generated by the Resnet101 network with embedded DCM module are input into the adaptive spatial feature fusion model ASFF, and ASFF performs upsampling or downsampling operations on P2 to P5.il , i = 2 to 5, so as to perform subsequent adaptive feature fusion. According to the characteristics of the feature maps at each level, the spatial weight vectors at each level are obtained.

[0099]

[0100] The feature maps at all levels are adaptively fused to obtain the feature maps P′2~P′5 at all levels. The formula is:

[0101]

[0102] The traditional FPN uses a layer-by-layer fusion method to perform each convolution stage C i The generated feature maps are integrated. Specifically, taking p2 and p3 as an example, before the p2 feature map is fused with p3, p3 itself has integrated the high-level feature information from p4 and p5. In this process, the feature maps of each layer are purely added layer by layer, resulting in the information of each layer being equally integrated into the final feature map, which may suppress some detailed information, externalize some unimportant information, and there are inconsistencies between different feature scales. In order to solve this problem, ASFF adds a learnable coefficient as a weight on the basis of FPN, automatically learns the relationship between each level, enables adaptive feature fusion, automatically adjusts the weight relationship between feature maps of each layer, improves the invariance of feature scale, and helps improve disease recognition performance.

[0103] The four feature maps P′2~P′5 obtained are further extracted through 3×3 convolution to obtain P″2~P″5. P″5 is downsampled again to generate a reference feature map P″6 that is more conducive to generating prediction results. Finally, P″2~P″6 are fused again as the output of the adaptive spatial feature fusion model ASFF, and input into the regional generation network RPN for target classification and boundary regression prediction. The extracted feature map is further input into RPN and RoI Align to generate a fixed-size feature region, and the corresponding mask, bounding box and category label are output using the full convolution layer and the fully connected layer.

[0104] In the process of training the bridge apparent slenderness disease detection model DCANet, a reasonable optimization method should be considered to accelerate the model convergence. Adam is used as the training optimizer, and stochastic gradient descent (SGD) is used for model training.

[0105] The loss function is used to guide the model training, and its formula is:

[0106] L=w cls L cls +w box L box +w maskL mask

[0107] Among them, L cls Represents the classification loss value, L box Represents the regression loss value of the bounding box; L mask Indicates the loss value of the masked part. cls 、w box 、w mask They represent the corresponding weight coefficients, and these weight coefficients are not equal. By assigning different loss function weights to different types of diseases, the difficult-to-detect parts such as the instance information of slender diseases can be more fully learned, thereby improving the sensitivity of the model.

[0108] In order to improve the training efficiency of the model, when training the bridge apparent slenderness disease detection model DCANet, the transfer learning method is first used to transfer the training weights of the COCO dataset to initialize the model; then, a phased training method is adopted, that is, first adjusting the parameters of the model head layer through training while freezing other layers, and then training all layers while unfreezing other layers.

[0109] S3: Use the trained bridge apparent slenderness defect detection model DCANet to automatically identify bridge apparent slenderness defects.

[0110] An embodiment of the present invention provides a device for identifying apparent slenderness defects of bridges based on a parallel convolution adaptive segmentation model, comprising one or more processors for implementing the method for identifying apparent slenderness defects of bridges based on a parallel convolution adaptive segmentation model in the above embodiment.

[0111] The embodiment of the bridge apparent slenderness disease identification method based on the parallel convolution adaptive segmentation model of the present invention can be applied to any device with data processing capability, and the device with data processing capability can be a device or apparatus such as a computer. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware.

[0112] Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capability in which it is located to read the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, the bridge apparent slenderness disease identification device based on the parallel convolution adaptive segmentation model of the present invention is a hardware structure of any device with data processing capability. In addition to the processor, memory, network interface, and non-volatile memory, any device with data processing capability in which the device in the embodiment is located can also include other hardware according to the actual function of the device with data processing capability, which will not be described in detail.

[0113] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0114] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0115] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the method for identifying apparent slenderness defects of bridges based on a parallel convolution adaptive segmentation model in the above embodiment is implemented.

[0116] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of any device with data processing capability and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0117] Example

[0118] Due to the special characteristics of bridge structures and the safety of shooting equipment, the defects in most images account for a low percentage of pixels in the image and the background complexity is high. In this example, 1,794 images of bridge surface slender defects were selected, including 294 exposed reinforcements and 1,500 cracks. All images are from engineering site images and do not include defect image data collected in laboratory environments. Considering the hardware resource cost and computational complexity of neural network data processing, we appropriately cropped the sample images into images with a resolution of 800*600 and annotated them using Labelme software. Figure 3The MSDefect dataset constructed in the embodiment is shown. The dataset is divided into non-overlapping training dataset and validation dataset, and each set of data uses the original image as input data and the annotated mask image as a prediction comparison value for testing and training network performance.

[0119] The bridge apparent disease images in the data set are used as the input data of the model for model training, with 500 iterations and 30 pre-trainings. If the index requirements are not met, the hyperparameter optimization is continued. If the index is met, the training weights are saved to obtain the DCANet bridge apparent slender disease detection model after training. In this model, the weights of most diseases are set to [1,1,1], while the weights of slender diseases are set to [0.5,0.5,1]. The relevant hyperparameters are determined by experiments, see Table 1. The ReLU activation parameter is set to 0.2, and training is performed for 200-500 epochs.

[0120] Table 1. DCANet model hyperparameter settings

[0121]

[0122] In this embodiment, IOU, precision P, recall rate R, F1 score and mean average precision mAP are used to evaluate the recognition performance of the bridge surface subtle damage detection model.

[0123] IOU (the Intersection over Union) represents the ratio of the overlapping area to the total area of ​​the actual area and the predicted area, and is used to evaluate the accuracy of instance segmentation. The higher the IoU value, the better the instance segmentation performance. The formula is:

[0124]

[0125] Among them, S overlap is the overlapping area between the actual area and the predicted area, S union is the total area of ​​the actual area and the predicted area.

[0126] Precision P refers to the proportion of true positive samples among all samples predicted to be positive, which is used to evaluate the model's ability to distinguish negative samples. Its calculation formula is:

[0127]

[0128] Among them, the positive samples predicted by the model as positive are TP (true positive); the negative samples predicted by the model as negative are TN (true negative); the negative samples predicted by the model as positive are FP (false positive); and the positive samples predicted by the model as negative are FN (false negative). Compared with classification tasks, instance segmentation has higher requirements, so the standard for correct identification is also higher. The detection is correct only when the defect type of the detection area is consistent with the ground truth area and the IOU between the two areas reaches at least a certain value.

[0129] The recall rate R is the proportion of actual positive samples that are correctly identified as positive samples, which is used to evaluate the model's ability to identify positive samples. Its calculation formula is:

[0130]

[0131] F1score is the harmonic mean of precision and recall, and is a comprehensive indicator for measuring the accuracy of model segmentation:

[0132]

[0133] The mean average precision (mAP) is a comprehensive measure of the overall accuracy of the model. After obtaining the values ​​of precision and recall, a precision-recall (PR) curve is plotted with precision as the vertical axis and recall as the horizontal axis. The area enclosed by the PR curve and the coordinate axis represents the AP (mean average precision) value of a specific defect type, expressed as

[0134]

[0135] mAP is the average of all class AP values, expressed as

[0136]

[0137] Where n represents the number of categories, AP i Represents the AP value of the i-th category. The bounding box mAP calculated with an IOU threshold of 0.5 is denoted as mAP bb , and the mAP of the mask calculated with an IOU threshold of 0.5 is expressed as mAP m .

[0138] like Figure 4As shown in the figure, DCANet can automatically identify various defect images well, and the defect recognition accuracy has been improved to a certain extent compared with the baseline. Taking cracks as an example, mAP increased from 5.26% to 34.92%, an increase of 563.54%; F1score increased from 7.17% to 34.12%, an increase of 375.69%; taking exposed reinforcement as an example, mAP increased from 0 to 67.49%; F1score increased from 0 to 50.85%. Through the model comparison experiment (see Table 2 for the comparison experiment configuration), it can be found that each module in DCANet has a positive impact on the model. When all modules are integrated, the best experimental results can be achieved. For the three models, the crack target was successfully detected, but the segmentation results of the baseline had most of the recognition missing or wrong recognition. As for AF, the locations of most defects can be detected, but the segmented masks are smaller than the ground truth or incorrect. DCA segmented more correct results and performed best among the three models. The segmentation results of exposed steel bars are similar to those of cracks. The baseline cannot generate segmentation results. AF can detect some instances, but there are incorrect segmentations. DCA successfully detects a relatively correct defect range. It can be found that DCANet can also perform well in different scenes with complex backgrounds and small proportions. In addition, it can be found that compared with DCA, more images of baseline and AF are concentrated in the parts with low thresholds and accuracy ( Figure 4 b cracks, Figure 4 c exposed ribs), which highlights the ability of DCA in handling complex geometric defect features.

[0139] Table 2 Comparative experimental settings

[0140]

[0141] At the same time, for different Dsconv positions, Figure 5The model recognition results under different Dsconv positions are shown in Figure 1. DSConv is inserted into different conv blocks or identity blocks in ResNet101, including low-level or high-level stages. At the same stage, conv blocks show better accuracy and more successful segmentation cases than identity blocks, which shows that inserting DSConv in conv blocks usually has better model performance than in identity blocks. This is because conv blocks are mainly used for feature perception, and identity blocks are mainly used for feature transfer. Accurate feature perception is crucial to improving the accuracy of segmentation results. Therefore, embedding DSConv into conv blocks is more conducive to optimizing model performance. For the same block, the low-level stage shows better accuracy and more successful detection image distribution than the high-level stage, which shows that inserting DSConv into the low-level stage usually shows better model performance than inserting it into the high-level stage. This is because the parameters of the low-level stage are more sensitive to the input and cause changes in the low-level features, which in turn affect the high-level features. Therefore, integrating DSConv into the low-level stage will significantly enhance the model performance.

[0142] Those skilled in the art can understand that the above are only preferred examples of the invention and are not intended to limit the invention. Although the invention is described in detail with reference to the above examples, those skilled in the art can still modify the technical solutions recorded in the above examples or replace some of the technical features therein with equivalents. Any modification, equivalent replacement, etc. made within the spirit and principle of the invention shall be included in the protection scope of the invention.

Claims

1. A bridge apparent slenderness defect recognition method based on a parallel convolution adaptive segmentation model, characterized in that: The method comprises the following steps: S1: Construct a dataset of bridge apparent slenderness defects; S2: Construct a bridge apparent slenderness disease detection model DCANet, which includes a feature extraction network with a built-in parallel feature extraction module DCM, an adaptive spatial feature fusion model ASFF, RPN, RoIAlign, and a network head architecture that outputs prediction results; the parallel feature extraction module DCM is a parallel module that integrates standard convolution Conv and dynamic snake convolution DSConv. The feature extraction of each stage is interspersed in parallel, where a single branch uses the dynamic snake convolution method to further perceive the morphological characteristics of the bridge apparent slenderness disease, and the other branch uses standard convolution to constrain global information; the parallel feature extraction module DCM is embedded in the identity block and convolution block of each stage of the feature extraction network; The adaptive spatial feature fusion model ASFF performs layer-by-layer superposition and fusion of the high semantic feature map generated by the last convolution stage of the feature extraction network and the feature maps generated by the remaining convolution stages except the first convolution stage output by the feature extraction network to obtain a multi-scale feature map Pi, i = 2~k; then performs an upsampling or downsampling operation F on Pi il , and according to the characteristics of the feature maps at all levels, the spatial weight vectors at all levels are obtained Finally, adaptively fuse the feature maps at all levels to obtain the feature maps P′i at all levels; then perform convolution on P′i to obtain P″i; downsample P″k to obtain P″(k+1), and fuse P″2 to P″(k+1) again as the output of the adaptive spatial feature fusion model ASFF; The dynamic snake convolution DSConv dynamically adjusts the convolution kernel through iterative offset so that it can better cover the shape of the disease and extract features from the disease image area. S3: Use the trained bridge apparent slenderness defect detection model DCANet to automatically identify bridge apparent slenderness defects.

2. The bridge apparent slenderness defect recognition method based on the parallel convolution adaptive segmentation model according to claim 1 is characterized in that: The execution steps of the parallel feature extraction module DCM are as follows: (1) Initializing the parallel convolution module in the parallel feature extraction module DCM; (2) Feature extraction is performed on the input disease feature map through standard convolution and dynamic snake convolution respectively. The parameters of dynamic snake convolution need to be continuously adjusted according to the image characteristics and model performance; (3) The feature maps generated by the two convolutions are integrated, and the output weights of the two convolutions are adjusted and optimized according to the model performance and image characteristics to ensure effective feature fusion and enhance the model's ability to recognize slender and complex diseases.

3. The bridge apparent slenderness defect recognition method based on the parallel convolution adaptive segmentation model according to claim 1 is characterized in that: The execution steps of the dynamic snake convolution DSConv are as follows: (1) Determine the center coordinate starting point K i±c =(x i±c ,y i±c ), c = {0, 1, 2, 3, 4}, c is the horizontal distance from the center grid, and the center coordinate starting point is used as the starting point of the convolution kernel iterative offset; (2) Select the next position of each target to be processed for observation in turn, so as to ensure the continuity of attention and not spread the perception range too far due to large deformation offset. i Initially, each grid K i±c The position depends on the previous grid K i±c-1 Position, that is, each grid K i±c In the previous grid K i±c-1 Add an offset Δ={δ|δ∈[-1,1]} to the basis, and the same applies to the y-axis direction; (3) If the convolution kernel sensing area formed by the point selected in step (2) contains part of the location of the disease, then select this point as the starting point of the next convolution iteration; otherwise, return to step (2) and continue to select a new direction for offset; (4) Smoothing the convolution kernel perception area selected in step (3) by bilinear interpolation; (5) Repeat steps (2) to (4) so ​​that the convolution kernel is iteratively shifted to a position that can better cover the shape of the disease, and perform feature extraction of the corresponding image area.

4. The bridge apparent slenderness defect recognition method based on the parallel convolution adaptive segmentation model according to claim 1 is characterized in that: The feature extraction network is Resnet101, which includes five convolution stages C1, C2, C3, C4, and C5. The specific workflow of the adaptive spatial feature fusion model ASFF is as follows: (1) The Resnet101 network with the embedded DCM module upsamples the feature map P5 generated in the C5 stage, and then superimposes and fuses it layer by layer with the feature maps generated by C2, C3, and C4 to obtain multi-scale feature maps P2, P3, and P4; (2) For the first level feature map output Pl, the remaining three feature maps are upsampled or downsampled F il , in order to perform subsequent adaptive feature fusion; (3) According to the characteristics of the feature maps at each level, the spatial weight vectors at each level are obtained (4) Adaptively fuse the feature maps at all levels according to the formula Obtain the characteristic maps P'2~P'5 at each level of scale; (5) During the training process, the weight vector is continuously optimized and updated to achieve adaptive fusion effect.

5. The bridge apparent slenderness defect recognition method based on the parallel convolution adaptive segmentation model according to claim 1 is characterized in that: The loss function of the bridge apparent slenderness defect detection model DCANet during training is the weighted sum of the classification loss value, the bounding box regression loss value and the mask loss value; the weight of the loss function of each category is different.

6. The bridge apparent slenderness defect recognition method based on the parallel convolution adaptive segmentation model according to claim 1 is characterized in that: When training the bridge apparent slenderness defect detection model DCANet, the model is first initialized by transferring the training weights of the COCO dataset using a transfer learning method. Then, a staged training method is adopted. That is, the parameters of the model head layer are first adjusted through training while the other layers are frozen. Then, all layers are trained while the other layers are unfrozen.

7. A bridge apparent slenderness defect recognition system based on a parallel convolution adaptive segmentation model, characterized by: The method comprises one or more processors for implementing the bridge apparent slenderness defect recognition method based on the parallel convolution adaptive segmentation model according to any one of claims 1 to 6.

8. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the electronic device, enables the electronic device to implement the bridge apparent slenderness defect identification method based on the parallel convolution adaptive segmentation model as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that A program is stored thereon, and when the program is executed by a processor, the method for identifying bridge apparent slenderness defects based on a parallel convolution adaptive segmentation model as claimed in any one of claims 1 to 6 is implemented.

Images