A double-branch network method for underwater dam crack detection

Abstract

The invention provides an underwater dam crack detection method of a double-branch network, comprising a CNN encoder and a Transformer encoder and the following steps: a fusion interaction module WIM interacts the multi-scale local features extracted by the CNN encoder with the global features of the Transformer encoder to enhance the extraction of effective features, and fuses the features at the end of the double-branch encoder and with an FLM module to achieve the effect of suppressing background noise information; secondly, a multi-scale gated adaptive fusion module MGIF can capture more complex dependencies and restore lost detail information by fusing feature maps of different levels between the encoder and the decoder, thereby finally achieving accurate extraction of crack texture features and improving the accuracy of prediction results.

Description

Technical Field

[0001] The invention relates to the field of underwater structural integrity detection, and in particular to an underwater dam crack detection method with a double-branch network. Background Art

[0002] Compared with the above-water part, the underwater dam body of the reservoir dam is more prone to cracks due to the combined effects of long-term immersion, temperature changes, water chemical corrosion and hydraulic fracturing. With the continued use of the dam, the number of cracks continues to increase, and some cracks may even extend into the interior of the dam, seriously endangering the integrity and bearing capacity of the structure. Therefore, underwater dam body crack detection is of great significance to maintain the normal operation of the structure and ensure its safety.

[0003] The attachment of suspended sediments and aquatic plants will increase the noise of the crack image on the dam surface and blur the crack texture morphology. Therefore, the accurate detection of cracks faces severe challenges. The development of deep learning provides a new way for crack detection. At present, image segmentation is the main technology for automatic crack detection, and convolutional neural network (CNN) has made significant progress in tasks such as target detection, image classification and semantic segmentation. However, convolution operations have limitations in encoding long-distance dependencies, and as the spatial distance increases, the computing resources required to calculate the correlation between positions increase exponentially. To solve this problem, the attention mechanism is introduced to enhance the useful features in the feature map. However, the limitations of convolution still restrict the further improvement of segmentation performance. In recent years, Transformer has been introduced into the field of computer vision. Although Transformer can calculate global dependencies through position encoding or attention mechanism, its computational complexity is significantly improved. In order to improve segmentation accuracy and reduce computational complexity, many studies have tried to balance segmentation accuracy and computational complexity by combining CNN and Transformer. However, in the detection of underwater dam cracks, the attachment of suspended sediments and aquatic plants causes the crack image to present a large amount of low-frequency and high-frequency noise interference, making it difficult for the convolution kernel to accurately extract and focus on the real crack features, especially the extraction of crack texture features. Summary of the invention

[0004] In view of the problems in the prior art, the present invention provides an underwater dam crack detection method based on a double-branch network, which aims to enhance the extraction of effective features, suppress background noise information, and improve the accuracy of extracting crack texture features.

[0005] A method for detecting cracks in underwater dams using a dual-branch network includes a CNN encoder and a Transformer encoder. The CNN encoder includes five layers of downsampling convolutional layers. When an image passes through the five layers of downsampling convolutional layers in sequence, the number of channels of the image's feature map doubles layer by layer, and the size of the image's feature map decreases layer by layer. The Transformer encoder includes four layers of Tansformer modules. When an image passes through the Tansformer modules in sequence for feature extraction, the size of the image's feature map decreases layer by layer to 1 / 4 of the size of the image's feature map. The method also includes the following steps:

[0006] Step 1: The dam crack image is sent to the CNN encoder and the Transformer encoder for feature extraction. Then, the outputs of the second to fourth layers of the CNN encoder are matched one by one with the outputs of the first to third layers of the Transformer encoder and both are fused and interacted through the fusion interaction module WIM to obtain the feature map C1;

[0007] Step 2: Return the feature map C1 all the way back to the corresponding layer of the Transformer encoder, then fuse the output of the fifth layer of the CNN encoder with the output of the fourth layer of the Transformer encoder through the feature reconstruction module FLM to obtain the feature map C2; at the same time, send the feature map C1 to the multi-scale gated adaptive fusion module MGIF. The multi-scale gated adaptive fusion module MGIF first fuses the feature maps of adjacent levels across scales step by step, and then fuses the corresponding level features through the adaptive gating mechanism to obtain the feature map ;

[0008] Step 3: Combine feature map C2 and feature map The decoder includes four upsampling convolution layers. When the feature map C2 is processed by the four upsampling convolution layers of the decoder in turn, the feature map The jump connection is made to the second to fourth layers of the decoder to restore the image feature information and obtain the detection results of the dam cracks.

[0009] Further: In step 2, the feature map C1 is:

[0010] ;

[0011] in, ∈R C×H×W , represents the initial feature integration, M represents the feature weight, A is the output of the second to fourth layers of the CNN encoder, and B is the output of the first to third layers of the Transformer encoder.

[0012] Further: Feature map for:

[0013]

[0014]

[0015]

[0016] in, are the embedding weight, gating weight and bias of the context embedding operator, respectively. is the feature map C1, C is the number of channels of the feature map C1, is the activation function.

[0017] Further: In step 4, the shape of each feature map in the output of the fifth layer of the CNN encoder and the output of the fourth layer of the Transformer encoder is adjusted from (B, C, H, W) to (B, C, H×W) by reshaping, and is divided into two branches. One branch is convolutionally fused by global average pooling and maximum pooling to achieve feature reconstruction. The result of feature reconstruction is successively subjected to the cross-attention mechanism and spatial feature fusion. The weighted result of spatial feature fusion will be multiplied with the output of another branch whose input feature is adjusted by reshaping, and then the result will be weightedly fused with the output of another reshape branch to generate a feature representative; then the feature representative generated by the output of the fifth layer of the CNN encoder is fused with the feature representative generated by the output of the fourth layer of the Transformer encoder to obtain feature map C2.

[0018] Further: Each feature map in the output of the fifth layer of the CNN encoder and the output of the fourth layer of the Transformer encoder is recorded as a feature map , the feature map output by the fifth layer of the CNN encoder is recorded as , the feature map output by the fourth layer of the Transformer encoder is recorded as , then the feature represents for:

[0019]

[0020]

[0021]

[0022]

[0023]

[0024] in, They represent global average pooling and maximum pooling respectively, and the value of i is n or m; n and m represent the subscripts of the output feature maps of the fifth layer of the CNN encoder and the fourth layer of the Transformer encoder respectively.

[0025] The beneficial effects of the present invention are as follows: through the CNN, WIM and FLM component feature reconstruction interactive encoder (WFL-Encoder), the different levels of features extracted by the CNN encoder are interactively fused with the features in the WIM and Transformer encoder to enhance the extraction of effective features, and then the features are fused and reconstructed through the FLM to suppress background noise information; and then through the multi-scale gated adaptive fusion module MGIF set between the encoder and the decoder, cross-level feature fusion between the encoder and the decoder is realized, which can capture more complex dependencies, and finally realize accurate extraction of crack texture features, thereby improving the accuracy of prediction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a flow chart of the present invention;

[0027] Figure 2 It is the flow chart of data processing in the feature reconstruction module FLM;

[0028] Figure 3 This is a flowchart of data processing in the multi-scale gated adaptive fusion module MGIF;

[0029] Figure 4 This is a diagram comparing the detection effects of different models on the DeepCrack dataset. DETAILED DESCRIPTION

[0030] The present invention is described in detail below in conjunction with the accompanying drawings. The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention. The directional terms such as left, middle, right, top, and bottom in the embodiments of the present invention are only relative concepts or are based on the normal use state of the product, and should not be considered as restrictive.

[0031] A double-branch network underwater dam crack detection method, such as Figure 1As shown, it includes a CNN encoder and a Transformer encoder. The CNN encoder includes five layers of downsampling convolution layers. When the image passes through the five layers of downsampling convolution layers in sequence, the number of channels of the image's feature map doubles layer by layer, and the size of the image's feature map decreases layer by layer. The Transformer encoder includes four layers of Tansformer modules. When the image passes through the Tansformer modules in sequence for feature extraction, the size of the image's feature map is reduced to 1 / 4 of its size layer by layer. It is characterized in that it also includes the following steps:

[0032] Step 1: The dam crack image is sent to the CNN encoder and the Transformer encoder for feature extraction. Then, the outputs of the second to fourth layers of the CNN encoder are matched one by one with the outputs of the first to third layers of the Transformer encoder and fused and interacted through the fusion interaction module WIM to obtain the feature map C1; the feature map C1 is:

[0033] ;

[0034] in, ∈R C×H×W , represents the initial feature integration, M represents the feature weight, A is the output of the second to fourth layers of the CNN encoder, and B is the output of the first to third layers of the Transformer encoder;

[0035] Step 2: Return the feature map C1 all the way back to the corresponding layer of the Transformer encoder, and then fuse the output of the fifth layer of the CNN encoder with the output of the fourth layer of the Transformer encoder through the feature reconstruction module FLM to obtain the feature map C2; After the above series of processing, the effective feature expression is enhanced, the useless redundant noise information is removed, and the correlation between different feature maps is captured. While reducing the computational complexity to a certain extent, rich spatial information is retained, the feature expression ability is significantly improved, and the robustness and generalization ability of the feature are enhanced; Figure 2As shown, specifically, the shape of each feature map in the output of the fifth layer of the CNN encoder and the output of the fourth layer of the Transformer encoder is adjusted from (B, C, H, W) to (B, C, H×W) by reshaping, and is divided into two branches. One branch is convolutionally fused by global average pooling and maximum pooling to achieve feature reconstruction. The result of feature reconstruction is successively subjected to the cross-attention mechanism and spatial feature fusion. The weighted result of spatial feature fusion will be multiplied with the output of the other branch whose shape is adjusted by reshaping the input feature, and then the result will be weightedly fused with the output of the other reshape branch to generate a feature representative; then the feature representative generated by the output of the fifth layer of the CNN encoder and the feature representative generated by the output of the fourth layer of the Transformer encoder are feature fused to obtain feature map C2; wherein, each feature map in the output of the fifth layer of the CNN encoder and the output of the fourth layer of the Transformer encoder is recorded as a feature map , the feature map output by the fifth layer of the CNN encoder is recorded as , the feature map output by the fourth layer of the Transformer encoder is recorded as , then the feature represents for:

[0036]

[0037]

[0038]

[0039]

[0040]

[0041] in, Represent global average pooling and maximum pooling respectively, i is n or m; n and m represent the subscripts of the fifth-layer output feature map of the CNN encoder and the fourth-layer output feature map of the Transformer encoder respectively; since the feature extraction capabilities of the CNN encoder and the Transformer encoder are naturally complementary, the fusion interaction module WIM implements weighting of the features of the CNN encoder through the global and local attention mechanisms, and performs weighted fusion, thereby interacting the multi-scale feature layer information to the Transformer encoder, enhancing the local perception ability of the Transformer encoder branch;

[0042] At the same time, the feature map C1 is sent to the multi-scale gated adaptive fusion module MGIF, as shown in Figure 3As shown in the figure, the multi-scale gated adaptive fusion module MGIF first fuses the feature maps of adjacent levels across scales step by step, and then fuses the corresponding level features through the adaptive gating mechanism to obtain the feature map ;

[0043] Feature Map for:

[0044]

[0045]

[0046]

[0047] in, are the embedding weight, gating weight and bias of the context embedding operator, respectively. is the feature map C1, C is the number of channels of the feature map C1, is the activation function; after being processed by the multi-scale gated adaptive fusion module MGIF, more complex dependencies can be captured through more efficient feature fusion between the encoder and decoder, and the normalization of the global context can be introduced; by adjusting the trainable parameters, the model can adaptively optimize the global features and enhance the feature expression ability. At the same time, due to the accumulation of information caused by multi-level fusion, repeated redundant information is removed by weight assignment. This setting is particularly suitable for small and unbalanced data sets, which not only reduces the computational burden caused by excessive fusion, but also improves the segmentation performance and generalization ability of the model;

[0048] Step 3: Combine feature map C2 and feature map The decoder includes four upsampling convolution layers. When the feature map C2 is processed by the four upsampling convolution layers of the decoder in turn, the feature map The jump connection is made to the second to fourth layers of the decoder to restore the image feature information and obtain the detection results of the dam cracks.

[0049] In the comparative experiment, as shown in the following table, the present invention compares the mIoU, Acc, Re, and F1 of different models on the DeepCrack dataset, and combines the attached Figure 4 As shown;

[0050] Models mIoU (%) Acc(%) Re(%) F1(%) SegNet 86.25 85.06 87.14 86.36 UNet 83.67 84.78 85.89 85.08 CarNet 82.51 87.37 86.75 85.67 DTrC-Net 85.37 85.68 87.14 86.24 Proposed 90.47 91.68 91.61 91.39

[0051] It can be seen that in the present invention, the fusion interaction module WIM interacts the multi-scale local features extracted by the CNN encoder with the global features of the Transformer encoder, enhances the extraction of effective features, and fuses the features at the end of the dual-branch encoder and with the FLM module, thereby achieving the effect of suppressing background noise information; secondly, the multi-scale gated adaptive fusion module MGIF restores the lost detail information by fusing feature maps of different levels between the encoder and the decoder; finally, through extensive experimental evaluation, compared with the performance of the current mainstream segmentation algorithm, the mIoU of the present invention reaches 90.47%, the Re is 91.61%, and the FLOPs is only 1.6G. It can be seen that the present invention has higher superiority.

[0052] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A method for detecting cracks in underwater dams using a dual-branch network, comprising a CNN encoder and a Transformer encoder, wherein the CNN encoder comprises five layers of downsampling convolutional layers, and when an image passes through the five layers of downsampling convolutional layers in sequence, the number of channels of the image's feature map doubles layer by layer, and the size of the image's feature map decreases layer by layer; the Transformer encoder comprises four layers of Tansformer modules, and when an image passes through the Tansformer modules in sequence for feature extraction, the size of the image's feature map decreases layer by layer to 1 / 4 thereof, and is characterized in that: The following steps are also included: Step 1: The dam crack image is sent to the CNN encoder and the Transformer encoder for feature extraction. Then, the outputs of the second to fourth layers of the CNN encoder are matched one by one with the outputs of the first to third layers of the Transformer encoder and weighted fusion interaction is performed through the fusion interaction module WIM to obtain the feature map C1. Step 2: Return the feature map C1 all the way back to the corresponding layer of the Transformer encoder, then fuse the output of the fifth layer of the CNN encoder with the output of the fourth layer of the Transformer encoder through the feature reconstruction module FLM to obtain the feature map C2; at the same time, send the feature map C1 to the multi-scale gated adaptive fusion module MGIF. The multi-scale gated adaptive fusion module MGIF first fuses the feature maps of adjacent levels across scales step by step, and then fuses the corresponding level features through the adaptive gating mechanism to obtain the feature map Step 3: Combine feature map C2 and feature map The decoder includes four upsampling convolution layers. When the feature map C2 is processed by the four upsampling convolution layers of the decoder in turn, the feature map The jump connection is made to the second to fourth layers of the decoder to restore the image feature information and obtain the detection results of the dam cracks.

2. The underwater dam crack detection method of the double-branch network according to claim 1 is characterized in that: In step 2, the feature map C1 is: Where C1∈R C×H×W , represents the initial feature integration, M represents the feature weight, A is the output of the second to fourth layers of the CNN encoder, and B is the output of the first to third layers of the Transformer encoder.

3. The underwater dam crack detection method of the double-branch network according to claim 1 is characterized in that: Feature Map for: Among them, u c 、v c and w c are the embedding weight, gating weight and bias of the context embedding operator, respectively, and x c is the feature map C1, C is the number of channels of the feature map C1, and tanh is the activation function.

4. The underwater dam crack detection method of the double-branch network according to claim 1 is characterized in that: In step 2, the shape of each feature map in the output of the fifth layer of the CNN encoder and the output of the fourth layer of the Transformer encoder is adjusted from (B, C, H, W) to (B, C, H×W) by reshaping, and is divided into two branches. One branch is convolutionally fused by global average pooling and maximum pooling to achieve feature reconstruction. The result of feature reconstruction is successively subjected to the cross attention mechanism and spatial feature fusion. The weighted result of spatial feature fusion will be multiplied with the output of the other branch whose input feature is adjusted by reshaping, and then the result will be weightedly fused with the output of the other reshape branch to generate a feature representative. Then, the feature representative generated by the output of the fifth layer of the CNN encoder is fused with the feature representative generated by the output of the fourth layer of the Transformer encoder to obtain feature map C2.

5. The underwater dam crack detection method of the double-branch network according to claim 4 is characterized in that: Each feature map in the output of the fifth layer of the CNN encoder and the output of the fourth layer of the Transformer encoder is denoted as feature map f i , the feature map output by the fifth layer of the CNN encoder is recorded as f n , the feature map output by the fourth layer of the Transformer encoder is recorded as f m , then the feature represents f′ i for: and i =avg_fc i +max_fc i Among them, avg_fc i , max_fc i They represent global average pooling and maximum pooling respectively, and the value of i is n or m; n and m represent the subscripts of the output feature maps of the fifth layer of the CNN encoder and the fourth layer of the Transformer encoder respectively.

Images