Underground pipeline anomaly detection method based on multispectral image collaboration

Abstract

The invention discloses an underground pipeline anomaly detection method based on multispectral image cooperation, and the method comprises the steps: firstly completing the fusion of three spectral images through employing an RGB three-color image fusion network based on multi-channel feature extraction; then, fusion of multispectral images is completed by using an infrared and RGB image fusion network based on a space / channel attention mechanism and a gradient aggregation residual dense block; and finally, an underground pipeline anomaly detection network of a double-branch encoder based on pyramid pooling visual Transformer (P2T) and a multi-scale network structure (Res2Net) is used to detect the anomaly of the underground pipeline. The multispectral image collaborative underground pipeline anomaly detection method provided by the invention can be used for detecting the anomaly in the underground pipeline. The infrared image data and the RGB image data in the pipeline are subjected to joint analysis, abnormal conditions and types in the pipeline are accurately identified, and the method has important practical application engineering value for completing underground pipeline internal state detection tasks.

Description

Technical Field

[0001] The present invention belongs to the technical field of underground pipeline internal anomaly detection, and in particular relates to a multi-spectral image coordinated underground pipeline anomaly detection method. Background Art

[0002] With the development of sensor technology, underground pipeline internal anomaly detection has entered the stage of multi-spectral collaboration. High-precision and high-stability sensors can better capture subtle changes inside the pipeline and provide more accurate basic data for subsequent data analysis. Research can be used to fuse multi-spectral images to complete underground pipeline anomaly detection.

[0003] The detection of underground pipeline anomalies in collaboration with multispectral images is generally a process of judging abnormal areas from RGB images based on the deep learning network structure model. Therefore, it is necessary to preprocess, extract features and identify RGB images first. This requires the use of digital image processing techniques, such as grayscale, edge detection, feature extraction, etc. In addition, the application of machine vision technology also provides a more efficient and accurate method for the processing and analysis of RGB images. However, the above algorithm only detects abnormal situations through RGB images. It is difficult to obtain sufficient information by relying on a single image, which makes it difficult to achieve accurate and efficient anomaly detection in complex underground areas. If the multispectral images are fused, more information inside the underground pipeline can be obtained. Therefore, how to better realize the fusion of multispectral images so as to improve the accuracy of subsequent detection of abnormal areas in underground pipelines is the key to the research. Summary of the invention

[0004] In order to solve the above problems, the present invention provides a multi-spectral image-coordinated underground pipeline anomaly detection method, which can accurately extract and fuse the RGB three-color image features, fuse the features of the infrared image and the RGB light image, better detect the abnormal state area, and improve the detection accuracy.

[0005] In view of the above-mentioned deficiencies in the prior art, the present invention provides a method for detecting anomalies of underground pipelines using multispectral images, comprising the following steps:

[0006] 1) Given a set of red images I R ∈H×W×1, blue image I B ∈H×W×1, green image I G ∈H×W×1, after a RGB three-color image fusion network based on multi-channel feature extraction, the fused RGB image I is finally obtained RGB ∈H×W×3;

[0007] 11) First, input the three color images into the feature extraction module to obtain the red, green and blue image feature maps after passing through the first stage of the network.

[0008] 12) After fusing these three features along the channel dimension, we get the fusion features of the first stage

[0009] 13) Continue to use the feature extraction module to extract the fusion features of the first stage Get refined features

[0010] 14) The first stage feature information obtained and the first stage refinement features Integrate and use the CBAM module to refine the feature residuals of each branch;

[0011] 15) After CBAM refinement, the information from the three branches is fused again along the channel dimension to produce the result of the second stage

[0012] 16) Combine standard convolution features and deformable convolution features to obtain more diverse fusion features F D ;

[0013] 17) By reintroducing the initial image I R , I B , I G To help reduce the deviation, the feature extraction module is used again to obtain the red, blue and green image features of the third stage

[0014] 18) The features of the three paths are fused and then subjected to a convolution operation to obtain the final result I RGB ;

[0015] 2) The obtained RGB image I RGB ∈H×W×3 and infrared image I ir ∈H×W×3, image fusion is achieved through feature extraction, aggregation and reconstruction, and the fused image I is obtained fu ∈H×W×3.

[0016] 21) First, a coarse feature extractor consisting of a convolutional layer and a LeakyReLU activation function is used to extract coarse features and shallow fusion features;

[0017] 22) Two GRXDB modules are deployed to enhance the ability to extract coarse texture features;

[0018] 23) The obtained feature map is input into the spatial / channel attention module (SCAM), and four convolution kernels of different sizes (1×1, 3×3, 5×5, 7×7) are used to expand the receptive field of the network;

[0019] 24) The output features are input into the spatial and channel attention masks to calculate the weights, and the pooling fusion method is used to generate the final refined feature map;

[0020] 25) Finally, the shallow features are connected to the final refined features and passed through the image reconstructor R I , reconstruct the fused image I fu ;

[0021] 3) The fused image I fu After passing through an underground pipeline anomaly detection network structure based on a dual-branch encoder of P2T and Res2Net, the edge refinement module (ERM) is finally used to generate the final abnormal area detection map P2.

[0022] 31) First, the CNN features obtained by the Res2Net module are filtered through the channel compression module (CCM), and the CNN output is obtained as

[0023] 32) Through the P2T module, the final output of Transformer is obtained as F t 1 , F t 2 ;

[0024] 33) The Transformer feature F will be obtained t 1 、F t 2 ,CNN features At the same time, it is input into the Fusion Enhancement Module (FEM) for fusion;

[0025] 34) The fused features are passed through the edge refinement module (ERM) to obtain edge features and refine the results to obtain the final abnormal area detection map P2.

[0026] Compared with the prior art, the beneficial effects are:

[0027] (1) A gradient-aggregated residual dense block (GRXDB) is designed. Two branches are integrated with the gradient operator to extract strong and weak textures. Various convolutional layers are fully utilized for feature extraction, dense connections are introduced in the main branch, and the Laplacian operator is integrated to further extract weak texture features. The individual outputs are combined using element-by-element addition to integrate deep features. The activation function of GRXDB is set to LeakyReLU to retain negative activation information, which can better meet the requirements of image fusion.

[0028] (2) This paper proposes an efficient fusion scheme, the Fusion Enhancement Module (FEM), to target the different characteristics of features extracted by CNN and Transformer, as well as the differences in semantic and detail features. Since the features extracted by Transformer contain more spatial and semantic information, Transformer can be used to obtain the spatial weights of features. Since the features extracted by CNN contain relevant information between all channels in the local space, CNN can be used to obtain channel size weights.

[0029] (3) The edge refinement module (ERM) is introduced to achieve edge noise filtering and edge detail information enhancement. The fused features are used for rough prediction, and the predicted features are used for dilation and erosion operations to obtain edge features. The edge features are used to refine the salient features, filter the noise, and increase the edge information. Edge noise filtering and edge detail information enhancement can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flow chart of the underground pipeline anomaly detection method coordinated by multi-spectral images of the present invention;

[0031] Figure 2 It is an RGB three-color image fusion network of the underground pipeline anomaly detection method based on multispectral image collaboration of the present invention;

[0032] Figure 3 A network structure diagram of infrared and RGB image fusion of the underground pipeline anomaly detection method based on multispectral image collaboration of the present invention;

[0033] Figure 4 The structure diagram of the GRXDB module proposed in the network of infrared and RGB image fusion of the underground pipeline anomaly detection method based on multispectral images of the present invention;

[0034] Figure 5 The SCAM module structure diagram proposed in the network of infrared and RGB image fusion of the underground pipeline anomaly detection method based on multi-spectral image collaboration of the present invention;

[0035] Figure 6It is a schematic diagram of the channel and spatial attention principle in the SCAM module structure of the underground pipeline anomaly detection method based on multi-spectral images of the present invention;

[0036] Figure 7 It is a network structure diagram of underground pipeline anomaly detection of the underground pipeline anomaly detection method coordinated by multi-spectral images of the present invention;

[0037] Figure 8 A FEM module structure diagram of an underground pipeline anomaly detection network of the underground pipeline anomaly detection method based on multi-spectral images of the present invention;

[0038] Fig. 9 The ERM module structure diagram of the underground pipeline anomaly detection network of the underground pipeline anomaly detection method coordinated with multi-spectral images of the present invention. DETAILED DESCRIPTION

[0039] The specific implementation of a multi-spectral image coordinated underground pipeline anomaly detection method of the present invention is described in detail below with reference to the accompanying drawings.

[0040] like Figure 1 As shown, the present invention provides a method for detecting underground pipeline anomalies using multispectral images, and the specific process is as follows:

[0041] 1) Given a set of red images I R ∈H×W×1, blue image I B ∈H×W×1, green image I G ∈H×W×1, after a RGB three-color image fusion network based on multi-channel feature extraction, the fused RGB image I is finally obtained RGB ∈H×W×3. Where H is the height of the image, W is the width of the image, 1 and 3 are the number of channels of the image, and the RGB three-color image fusion network is as follows Figure 2 shown.

[0042] 11) First, the three color images are input into the feature extraction module, which consists of a convolution layer, a regularization layer, and a PReLU activation function, which can be expressed as follows:

[0043]

[0044] represents the feature map extracted from the red image after passing through the first stage of the network, represents the feature map extracted from the blue image after passing through the first stage of the network, represents the feature map extracted from the green image after passing through the first stage of the network, PLU represents the PReLU activation function, Drop represents the regularization operation, and Conv(·) represents the convolution operation;

[0045] 12) After fusing these three features along the channel dimension, we get the fusion features of the first stage

[0046]

[0047] represents feature fusion operation;

[0048] 13) Continue to use the feature extraction module consisting of a convolution layer, a regularization layer, and a PReLU activation function to continue extracting the fusion features of the first stage get The process can be represented as follows:

[0049]

[0050] Refine features for the first stage;

[0051] 14) The first stage feature information obtained and the first stage refinement features The feature residuals of each branch are integrated and refined using the CBAM module. The process is expressed as follows:

[0052]

[0053] CABM stands for CABM module. After the red image features are refined by CBAM, After CBAM refinement of green image features, The blue image features refined by CBAM, It is the element addition operation;

[0054] 15) After CBAM refinement, the information from the three branches is fused again along the channel dimension to produce the result of the second stage

[0055]

[0056] 16) Combine standard convolution features and deformable convolution features to obtain more diverse fusion features F D , the process is expressed as follows:

[0057]

[0058] def is a deformable convolution operation;

[0059] 17) By reintroducing the initial image I R , I B , I GTo help reduce the deviation, the segmentation feature processing method of the residual enhancement module is used again to obtain the features of the third stage. It is represented by:

[0060]

[0061] is the red image feature of the third stage, is the blue image feature of the third stage, is the green image feature of the third stage;

[0062] 18) The features of the three paths are fused. After this fusion, a convolution operation is performed to obtain the final result:

[0063]

[0064] I RGB is the final fusion output graph;

[0065] 2) The obtained RGB image I RGB ∈H×W×3 and infrared image I ir ∈H×W×3, image fusion is achieved through feature extraction, aggregation and reconstruction, and the fused image I is obtained fu ∈H×W×3. The network structure diagram of infrared and RGB image fusion is as follows Figure 3 shown.

[0066] 21) First, a coarse feature extractor F = [·] consisting of a convolutional layer and a LeakyReLU activation function is used to extract coarse features and shallow fusion features:

[0067] F RGB =F[I RGB ]

[0068] F ir =F[I ir ]

[0069] F qc =F[C(I RGB ,I ir )] (9)

[0070] F RGB is the coarse feature of RGB image, F ir is the infrared image coarse feature, F qc is a shallow fusion feature, and C(·) represents a concatenation operation;

[0071] 22) The GRXDB module is deployed to enhance the ability to extract coarse texture features. The GRXDB module structure is shown in the figure. Figure 4 As shown;

[0072] The present invention designs a gradient aggregated residual dense block (GRXDB). The GRXDB module consists of three branches. First, the main branch integrates the Laplacian operator to further extract weak texture features, and then passes through four 3×3 convolution blocks (consisting of 3×3 convolution layers and LReLU functions) and one 1×1 convolution block (consisting of Concat function and 1×1 convolution layer). The residual branch consists of two 3×3 convolution blocks (consisting of 3×3 convolution layers and LReLU functions), a 1×1 convolution block (consisting of 1×1 convolution layers and LReLU functions) and a Sobel operator to retain strong texture features. The third branch remains unchanged as a residual branch that retains input feature information. Finally, the outputs of the main branch, residual branch and third branch are combined using element-by-element addition to integrate deep features. In order to better meet the requirements of image fusion, the present invention sets the activation function of GRXDB to LeakyReLU to retain negative activation information.

[0073] Two GRXDB modules are deployed to enhance the ability to extract coarse feature texture features, which can be expressed as follows:

[0074]

[0075] in, represents the addition of elements, and are the standard deviations of the Sobel gradient and Laplace gradient operators, respectively. GRXDB refers to the GRXDB module, and Conv(·) represents the convolution operation. RGB 1 represents the RGB feature map obtained after the first GRXDB module, F ir 1 Indicates the infrared feature map obtained after passing through the first GRXDB module. RGB 2 represents the RGB feature map obtained after the second GRXDB module, F ir 2 It represents the infrared feature map obtained after passing through the second GRXDB module;

[0076] 23) Input the obtained feature map into the spatial / channel attention module (SCAM). The SCAM module structure is shown in the following figure: Figure 5 As shown;

[0077] This paper proposes a lightweight and trainable multi-scale spatial / channel attention refinement module, which is called SCAM. First, four convolutional blocks with different kernel sizes (1×1, 3×3, 5×5, 7×7) are used to capture deep features with multiple receptive fields and scales. The activation function of these convolutional blocks is still LeakyReLU. Then, the obtained deep features are concatenated along the channel axis, and then sent to the spatial and channel attention layers by 1×1 convolutional blocks to adjust the number of channels. The generation of attention masks Ms and Mc is a key process in the attention mechanism, which aims to enhance the information capture ability of the network and obtain more accurate spatial and semantic information. After the attention branch is refined, upsampling is used to restore the channel feature map to its original size, and convolution operations are used to control the channel dimension of the spatial feature map. In order to generate feature maps that meet the requirements of both intra-class similarity and inter-class diversity, the pooling fusion block (PFB) is used to fuse the generated spatial and channel attention feature maps, thereby generating a fused feature map that meets the required conditions.

[0078] Before calculating the attention weights, four convolution kernels of different sizes (1×1, 3×3, 5×5, 7×7) are used to expand the receptive field of the network. The process can be expressed as follows:

[0079] F cout =Conv(C(Conv4(C(F ir 2 ,F RGB 2 )))) (11)

[0080] C(·) represents the concatenation operation, n represents the size of the convolution layer, and Conv4 represents the use of four convolution kernels of different sizes (1×1, 3×3, 5×5, 7×7) for convolution operations. cout The infrared and RGB image fusion feature map is obtained;

[0081] 24) Input the output features into the spatial and channel attention masks to calculate the weights. The principle diagram of channel and spatial attention is as follows Figure 6 As shown;

[0082] To generate the channel attention mask, global average pooling is first performed on each channel of the feature map to obtain the average value of all pixels within that channel. This process generates a new 1×1 channel map. Then, the Tanh activation function and a 1×1 convolution block are used to adjust the values ​​of the feature map. It is worth noting that the calculated value of the Tanh activation function is divided by 2 and added to 0.5, which maps the values ​​to [0,1]. Finally, the calculated channel map is multiplied with the input feature map to generate the output of the channel attention branch. Similarly, to obtain the spatial attention mask, a 1×1 convolution block is first used to reduce the channel dimension of the input feature map. Then, max pooling and average pooling are used simultaneously to obtain two feature maps, each with one channel and the same size as the input feature map. Next, the two feature maps are concatenated and their channel dimension is reduced to 1 using a 1×1 convolution block. The Tanh activation function is applied to this convolution block, consistent with the channel attention mechanism. Finally, the generated spatial attention mask is multiplied with the input feature map to obtain the output of the spatial attention branch.

[0083] Finally, the pooling fusion method is used to generate the final refined feature map. The process is expressed as follows:

[0084] F SC =PFB(Conv(F cout ⊙B c (M s )),Up(F cout ⊙B s (M c ))) (12)

[0085] M c and M s denote the channel attention mask and the spatial attention mask, respectively, and ⊙ denotes element-wise multiplication; B s (·) represents broadcasting the channel attention value along the spatial dimension, B c (·) indicates broadcasting the spatial attention value along the channel dimension; Up indicates the upsampling operation, F SC To refine the feature map, PFB is a pooling fusion operation;

[0086] 25) Finally, the shallow features are connected to the final refined features and passed through the image reconstructor R I , reconstruct the fused image I fu :

[0087] I fu =R I (C(F qc ,F SC )) (13)

[0088] I fu is the final fused image, RI Reconstruction operation for the image.

[0089] 3) The fused image I fu After a dual-branch encoder based on P2T and Res2Net, the underground pipeline anomaly detection network structure is finally used to generate the final abnormal area detection map P2 using the edge refinement module (ERM). The underground pipeline anomaly detection network structure is shown in the figure. Figure 7 shown.

[0090] 31) First, the CNN features obtained by the Res2Net module are filtered through the channel compression module (CCM), and the CNN output is obtained as

[0091] 32) Through the P2T module, the final output of Transformer is obtained as F t 1 , F t 2 ;

[0092] 33) The Transformer feature F will be obtained t 1 、F t 2 ,CNN features At the same time, it is input into the fusion enhancement module (FEM) for fusion. The structure diagram of the FEM module is as follows Figure 8 As shown;

[0093] Aiming at the different characteristics of the features extracted by CNN and Transformer, as well as the differences in semantic and detail features, the present invention proposes an efficient fusion scheme, the fusion enhancement module (FEM). Since the features extracted by Transformer contain more spatial and semantic information, Transformer can be used to obtain the spatial weight of the features. Subsequently, it is used to weight the CNN features in the spatial dimension and filter out the important areas. In addition, the features extracted by CNN contain relevant information between all channels in the local space, so CNN can be used to obtain the channel size weights. Then, it is used to weight the Transformer features in the channel dimension to filter out the more important channels. The useless features are filtered by mutual weighting to reduce the difference between the two branches, and then the fused features of the first step are obtained by concatenation and fusion. In addition, the fused features are spliced ​​and fused for the second time into a dual-branch weighted filter feature. Finally, the features of the two branches are not much different, and the features of the second dual-branch fusion step are directly spliced ​​to complete the third fusion step. Therefore, in the FEM module, the fused features of this layer are finally combined with the semantic features extracted by Transformer from the previous layer. The dependency between semantic information and detail information is established through self-attention. In the fusion enhancement module, it is first upsampled to generate the spatial attention weight X′.

[0094] X'=σ(Conv(ConCat(Mean(up(F t 2 )),Max(up(F t 2 ))))) (14)

[0095] Up represents the upsampling operation, Mean represents the average value operation on each channel, Max represents the maximum value operation on each channel value, Conv represents the convolution operation, ConCat represents the feature concatenation operation, and σ represents the sigmoid activation function. Then, using Generate channel attention weights, which can be expressed as

[0096]

[0097] Among them, Y′ is the channel attention weight, avgpool is the global average pooling operation, maxpool represents the maximum pooling operation, and Conv represents the convolution operation. The first weighted fusion step is performed after obtaining the corresponding weights. The specific process is as follows:

[0098]

[0099] M is the result of the first fusion stage, ⊙ represents element-wise multiplication, C brRepresents a convolution operation as well as normalization and ReLU operations (Conv-BN-ReLU). The weighted fused features are further fused with the original features to reduce the difference between the two branches, completing the second stage of fusion:

[0100]

[0101] M tf For M and F t 2 The fusion result, M cf M and The result of fusion; after two fusions to reduce the feature differences between the two branches, the third fusion step is performed to directly splice and merge the features of the two branches, so that the final feature F zz Contains rich details extracted by CNN and semantic information extracted by Transformer;

[0102] F zz =C br (ConCat(M tf ,M cf )) (18)

[0103] Finally, the feature F extracted from the Transformer of the current layer t 1 Through the self-attention mechanism and F zz Further fusion is performed to increase the global semantic information and obtain the output feature Z1.

[0104] Z1=Att(Linear(ConCat(Unfold(F zz ),F t 1 ))) (19)

[0105] Unfold represents the feature flattening operation, Linear represents the linear connection operation, and Att represents the self-attention calculation.

[0106] 34) The fusion features are passed through the edge refinement module (ERM) to obtain edge features and refine the results to obtain the final abnormal area detection map P2. The ERM module structure diagram is shown in Fig. 9 As shown;

[0107] First, three layers of Conv-BN-ReLU are superimposed to further extract input features, and then the convolution layer is used to obtain a rough prediction feature P1, which is described as:

[0108] F=C br (C br (C br (Z1)))

[0109] P1=Conv(F) (20)

[0110] Then, by dilating and corroding the rough prediction feature P1, the edge feature E is obtained:

[0111] E=Conv(dilate(P1)-erode(P1)) (21)

[0112] dilate represents the morphological expansion operation, and erode represents the morphological erosion operation. The edge feature E and the feature F are multiplied and added pixel by pixel to enrich the detail features while filtering out the noise to obtain a more refined feature F′:

[0113] F′=F+(E⊙F) (22)

[0114] The refined features are expanded and input into the Transformer layer to establish the dependency between the semantic and detail information in the features, and then pass through a convolutional layer to obtain the final detection result P2:

[0115] P2=Conv(TF(Unfold(F′))) (23)

[0116] TF stands for Transformer encoding operation.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention and are not intended to limit the present invention. A person skilled in the art should understand that the specific implementation of the present invention can be modified or replaced by equivalents, but these modifications or changes are within the scope of protection of the pending claims.

Claims

1. A multi-spectral image collaborative underground pipeline anomaly detection method, characterized in that: The underground pipeline anomaly detection method comprises the following steps: 1) Given a set of red images I R ∈H×W×1, blue image I B ∈H×W×1, green image I G ∈H×W×1, after a RGB three-color image fusion network based on multi-channel feature extraction, the fused RGB image I is finally obtained RGB ∈H×W×3, where H is the height of the image, W is the width of the image, and 1 and 3 are the number of channels of the image; 2) The obtained RGB image I RGB ∈H×W×3 and infrared image I ir ∈H×W×3, image fusion is achieved through feature extraction, aggregation and reconstruction, and the fused image I is obtained fu ∈H×W×3; 3) The fused image I fu After passing through an underground pipeline anomaly detection network with a dual-branch encoder based on P2T and Res2Net, the edge refinement module (ERM) is finally used to generate the final abnormal area detection map P2.

2. The method for detecting anomalies of underground pipelines using multispectral images according to claim 1, characterized in that: Step 1) comprises the following steps: 11) First, the three color images are input into the feature extraction module, which consists of a convolution layer, a regularization layer, and a PReLU activation function, which is expressed as follows: represents the feature map extracted from the red image after passing through the first stage of the network, represents the feature map extracted from the blue image after passing through the first stage of the network, represents the feature map extracted from the green image after passing through the first stage of the network, PLU represents the PReLU activation function, Drop represents the regularization operation, and Conv(·) represents the convolution operation; 12) After fusing these three features along the channel dimension, we get the fusion features of the first stage represents the feature fusion operation; 13) Continue to use the feature extraction module consisting of a convolution layer, a regularization layer and a PReLU activation function to extract the fusion features of the first stage get The process is represented as follows: Refine features for the first stage; 14) The first stage feature information obtained and the first stage refinement features The feature residuals of each branch are integrated and refined using the CBAM module. The process is expressed as follows: CABM stands for CABM module; After the red image features are refined by CBAM, After the green image features are refined by CBAM, The blue image features refined by CBAM, It is the element addition operation; 15) After CBAM refinement, the information from the three branches is fused again along the channel dimension to produce the result of the second stage 16) Combine standard convolutional features and deformable convolutional features to obtain more diverse fusion features F D , the process is expressed as follows: def is a deformable convolution operation; 17) By reintroducing the initial image I R , I B , I G To help reduce the deviation, the segmentation feature processing method of the residual enhancement module is used again to obtain the features of the third stage. It is represented by: is the red image feature of the third stage, is the blue image feature of the third stage, is the green image feature of the third stage; 18) The features of the three paths are fused. After this fusion, a convolution operation is performed to obtain the final result: I RGB This is the final fusion output image.

3. The method for detecting anomalies of underground pipelines using multispectral images according to claim 1, characterized in that: Step 2) comprises the following steps: 21) First, a coarse feature extractor F = [ ] consisting of a convolutional layer and a Leaky ReLU activation function is used to extract coarse features and shallow fusion features: F RGB =F[I RGB ] F ir =F[I ir ] F qc =F[C(I RGB ,I ir )] (9) F RGB is the coarse feature of RGB image, F ir is the infrared image coarse feature, F qc is a shallow fusion feature, and C(·) represents a concatenation operation; 22) Two GRXDB modules are deployed to enhance the ability to extract coarse feature texture features, which are expressed as follows: in, represents the addition of elements, and are the standard deviations of the Sobel gradient and Laplace gradient operators, respectively. GRXDB refers to the GRXDB module. Conv(·) represents the convolution operation. F RGB 1 represents the RGB feature map obtained after the first GRXDB module, F ir 1 represents the infrared feature map obtained after passing through the first GRXDB module, F RGB 2 represents the RGB feature map obtained after the second GRXDB module, F ir 2 It represents the infrared feature map obtained after passing through the second GRXDB module; 23) The obtained feature map is input into the spatial / channel attention module (SCAM). Before calculating the attention weight, four convolution kernels of different sizes (1×1, 3×3, 5×5, 7×7) are used to expand the receptive field of the network. The process is expressed as follows: F cout =Conv(C(Conv4(C(F ir 2 ,F RGB 2 )))) (11) C(·) represents the concatenation operation, Conv4 represents the convolution operation using four convolution kernels of different sizes (1×1, 3×3, 5×5, 7×7), and F cout The infrared and RGB image fusion feature map is obtained; 24) The output features are input into the spatial and channel attention masks to calculate the weights. Finally, the pooling fusion method is used to generate the final refined feature map. The process is expressed as follows: F SC =PFB(Conv(F cout ⊙B c (M s )),Up(F cout ⊙B s (M c ))) (12) M c and M s denote the channel attention mask and the spatial attention mask, respectively, and ⊙ denotes element-wise multiplication; B s (·) represents broadcasting the channel attention value along the spatial dimension, B c (·) indicates broadcasting the spatial attention value along the channel dimension, Up indicates the upsampling operation, and F SC To finally refine the feature map, PFB is a pooling fusion operation; 25) Finally, the shallow features are connected to the final refined features and passed through the image reconstructor R I , reconstruct the fused image I fu : I fu =R I (C(F qc ,F SC )) (13) I fu is the final fused image, R I Reconstruction operation for the image.

4. The method for detecting anomalies of underground pipelines using multispectral images according to claim 1, characterized in that: Step 3) comprises the following steps: 31) First, the CNN features obtained by the Res2Net module are filtered through the channel compression module (CCM), and the CNN output is obtained as F c 1 ; 32) Through the P2T module, the final output of Transformer is obtained as F t 1 , F t 2 ; 33) The Transformer feature F will be obtained t 1 、F t 2 , CNN feature F c 1 At the same time, it is input into the Fusion Enhancement Module (FEM) for fusion; 34) The fused features are passed through the edge refinement module (ERM) to obtain edge features and refine the results to obtain the final abnormal area detection map P2.

5. The method for detecting anomalies of underground pipelines using multispectral images according to claim 3 is characterized in that: The GRXDB module in step 22) is a gradient aggregation residual dense block, and the GRXDB module consists of three branches: a main branch, a residual branch and a third branch; The GRXDB module working process includes the following steps: First, the main branch integrates the Laplacian operator to further extract weak texture features, and then passes through four 3×3 convolution blocks and one 1×1 convolution block; the residual branch consists of two 3×3 convolution blocks, one 1×1 convolution block and the Sobel operator to retain strong texture features; the third branch remains unchanged as a branch that retains input feature information; finally, the outputs of the main branch, the residual branch and the third branch are combined using element-by-element addition to integrate deep features; the activation function of the GRXDB module is set to LeakyReLU to retain negative activation information.

6. The method for detecting anomalies of underground pipelines using multispectral images according to claim 3, characterized in that: Step 23) comprises the following steps: First, four convolution blocks with different kernel sizes (1×1, 3×3, 5×5, 7×7) are used to capture deep features with multiple receptive fields and scales; the activation function of these convolution blocks is still Leaky ReLU; then, the obtained deep features are connected along the channel axis, and then the number of channels is adjusted through a 1×1 convolution block, and then sent to the spatial and channel attention layers; the generation of attention masks Ms and Mc is a key process in the attention mechanism, which aims to enhance the information capture ability of the network and obtain more accurate spatial and semantic information; after refining the attention branch, upsampling is used to restore the channel feature map to its original size, and convolution operations are used to control the channel dimension of the spatial feature map; in order to generate feature maps that meet the requirements of both intra-class similarity and inter-class diversity, the pooling fusion block (PFB) is used to fuse the generated spatial and channel attention feature maps, thereby generating a fused feature map that meets the required conditions.

7. The method for detecting anomalies of underground pipelines using multispectral images according to claim 3, characterized in that: Step 24) comprises the following steps: The steps for generating the channel attention mask are as follows: First, global average pooling is performed on each channel of the feature map to obtain the average value of all pixels within the channel; this process generates a new 1×1 channel map; then, the Tanh activation function and a 1×1 convolution block are used to adjust the value of the feature map; it is worth noting that the calculated value of the Tanh activation function is divided by 2 and added to 0.5, which maps the value to [0,1]; finally, the calculated channel map is multiplied with the input feature map to generate the output of the channel attention branch; The steps for generating the spatial attention mask are as follows: first, a 1×1 convolution block is used to reduce the channel dimension of the input feature map; then, maximum pooling and average pooling are used simultaneously to obtain two feature maps, each with one channel and the same size as the input feature map; next, the two feature maps are concatenated and their channel dimension is reduced to 1 using a 1×1 convolution block; the Tanh activation function is applied to the convolution block, consistent with the channel attention mechanism; finally, the generated spatial attention mask is multiplied with the input feature map to obtain the output of the spatial attention branch.

8. The method for detecting anomalies of underground pipelines using multispectral images according to claim 4, characterized in that: The fusion enhancement module in step 33) includes the following steps: Transformer is used to obtain the spatial weight of the feature; subsequently, it is used to weight the CNN features in the spatial dimension and filter out the important areas; in addition, the features extracted by CNN contain the relevant information between all channels in the local space, so CNN is used to obtain the channel size weight; then, it is used to weight the Transformer features in the channel dimension to filter out the more important channels; the useless features are filtered by mutual weighting to reduce the difference between the two branches, and then the fused features of the first step are obtained by concatenation and fusion; in addition, the fused features are spliced ​​and fused for the second time into the dual-branch weighted filtering features; finally, the features of the two branches are not much different, and the features of the second dual-branch fusion step are directly spliced ​​to complete the third fusion step; therefore, in the FEM module, the fused features of this layer are finally combined with the semantic features extracted by Transformer from the previous layer; the dependency between the semantic information and the detail information is established by self-attention; in the fusion enhancement module, it is first upsampled to generate the spatial attention weight X′: X′=σ(Conv(ConCat(Mean(up(F t 2 )),Max(up(F t 2 ))))) (14) Up represents the upsampling operation, Mean represents the average value operation on each channel, Max represents the maximum value operation on each channel value, Conv represents the convolution operation, ConCat represents the feature concatenation operation, and σ represents the sigmoid activation function. Then, using F c 1 Generate channel attention weights, expressed as Y′=σ(Conv(avgpool(F c 1 ))+Conv(maxpool(F c 1 ))) (15) Among them, Y′ is the channel attention weight, avgpool is the global average pooling operation, maxpool represents the maximum pooling operation, and Conv represents the convolution operation. The first weighted fusion step is performed after obtaining the corresponding weights. The specific process is as follows: M=Conv(ConCat((Y′⊙F t 2 ),(X′⊙F c 1 ))) (16) M is the result of the first fusion stage, ⊙ represents element-wise multiplication, C br Represents a convolution operation as well as normalization and ReLU operations (Conv-BN-ReLU); the weighted fused features are further fused with the original features to reduce the difference between the two branches, completing the second stage of fusion: M tf =C br (ConCat(M,F t 2 )) M cf =C br (ConCat(M,F c 1 )) (17) M tf For M and F t 2 The fusion result, M cf For M and F c 1 The result of fusion; after two fusions to reduce the feature differences between the two branches, the third fusion step is performed to directly splice and merge the features of the two branches, so that the final feature F zz Contains rich details extracted by CNN and semantic information extracted by Transformer; F zz =C br (ConCat(M tf ,M cf )) (18) Finally, the feature F extracted from the Transformer of the current layer t 1 Through the self-attention mechanism and F zz Further fusion to increase the global semantic information and obtain the output feature Z1; Z1=Att(Linear(ConCat(Unfold(F zz ),F t 1 ))) (19) Unfold represents the feature flattening operation, Linear represents the linear connection operation, and Att represents the self-attention calculation.

9. The method for detecting anomalies of underground pipelines using multispectral images according to claim 8, characterized in that: Step 34) comprises the following steps: First, three layers of Conv-BN-ReLU are superimposed to further extract input features, and then the convolution layer is used to obtain a rough prediction feature P1, which is described as: F=C br (C br (C br (Z1))) P1=Conv(F) (20) Then, by dilating and corroding the rough prediction feature P1, the edge feature E is obtained: E=Conv(dilate(P1)-erode(P1)) (21) dilate represents the morphological expansion operation, and erode represents the morphological erosion operation; the edge feature E and the feature F are multiplied and added pixel by pixel to enrich the detail features and filter out the noise to obtain a finer feature F′: F′=F+(E⊙F) (22) The refined features are expanded and input into the Transformer layer to establish the dependency between the semantic and detail information in the features, and then pass through a convolutional layer to obtain the final detection result P2: P2=Conv(TF(Unfold(F′))) (23) TF stands for Transformer encoding operation.

Images