Leakage analysis method based on multi-modal image fusion

Abstract

The invention relates to the technical field of data processing, and discloses a leakage analysis method based on multi-modal image fusion, and the method comprises the steps: carrying out the feature extraction of an RGB image through employing a MobileNetV3 network, obtaining the RGB features, and carrying out the feature extraction of an infrared image through employing another MobileNetV3 network, and obtaining the features of the infrared image; and then the RGB features and the infrared image features are decoded through a decoder to generate a segmentation image. According to the method, the problems of low accuracy, low robustness and the like in the prior art are solved.

Description

Technical Field

[0001] The invention relates to the technical field of data processing, and in particular to a leakage analysis method based on multimodal image fusion. Background Art

[0002] In the environment of hydropower station cushion pond corridors, accurate identification and analysis of leakage is crucial to assess material quality and structural integrity. Although traditional deep learning models based on RGB images work well in most cases, the segmentation accuracy and robustness of these models are often challenged in environments with large lighting changes and complex textures. Since these models mainly rely on surface color information, their performance in complex environments is limited. Summary of the invention

[0003] In order to overcome the shortcomings of the prior art, the present invention provides a leakage analysis method based on multimodal image fusion to solve the problems of low accuracy and low robustness in the prior art.

[0004] The technical solution adopted by the present invention to solve the above problems is:

[0005] A leakage analysis method based on multimodal image fusion uses a MobileNetV3 network to extract features from RGB images to obtain RGB features, and uses another MobileNetV3 network to extract features from infrared images to obtain infrared image features; then the RGB features and infrared image features are decoded through a decoder to generate a segmentation map.

[0006] As a preferred technical solution, feature fusion and feature enhancement are performed on RGB features and infrared image features through a feature fusion enhancement module.

[0007] As a preferred technical solution, a multi-head self-attention mechanism is used to process RGB features and infrared image features, including the following steps:

[0008] D1, linearly transforms each input to prepare representation for each head, the formula is:

[0009]

[0010] In the formula, A i represents the query after linear transformation, and A represents the query before linear transformation. represents the query weight matrix, B i represents the key after linear transformation, B represents the key before linear transformation, represents the key weight matrix, C i represents the value after linear transformation, C represents the value before linear transformation, represents the value weight matrix;

[0011] D2 is calculated and scaled by the attention weights, and the formula is:

[0012]

[0013] In the formula, Attention(A i ,B i ,C i ) represents the focus filter, d m represents the dimension of B;

[0014] D3 combines the outputs of all heads and obtains the final output through linear transformation. The formula is:

[0015] MultiHead(A,B,C)=Concat(head1,…,head h ) O

[0016] In the formula, MultiHead(A,B,C) represents the final output, Concat(head1,…,head h ) represents the output of all heads after merging, U O represents the linear transformation matrix, h represents the number of heads;

[0017] Among them, the output head of each head i For the independently calculated Attention (A i ,B i ,C i ).

[0018] As a preferred technical solution, an ARM module is used to aggregate RGB features and infrared image features.

[0019] As a preferred technical solution, a cross-attention module is used to process RGB features and infrared image features: first, the features are refined, and the feature expression is enhanced in the spatial dimension and channel dimension through the cross-attention mechanism; then, the input features from different sources are fused through the feature fusion strategy to generate an optimized intermediate feature map.

[0020] As a preferred technical solution, the feature refinement process includes the following steps:

[0021] X1, with dimension R B×C×W×H The input feature X is transformed;

[0022] Among them, B is the batch size, C is the number of channels, W is the width of the feature map, H is the height of the feature map, and r is the reduction ratio, including the following transformations:

[0023] Key conversion:

[0024] Query conversion:

[0025] Value conversion: V = value_conv(X)∈R B×C×W×H ;

[0026] Where K represents the key, key_conv(·) represents the key conversion function, R represents the attention map, Q represents the query, query_conv(·) represents the query conversion function, V represents the value, and value_conv(·) represents the value conversion function;

[0027] X2, calculate the attention weights and rearrange them:

[0028]

[0029] Where Q′ represents the query tensor after flattening and transposition, view(·) represents reshaping the tensor, permute(·) represents changing the order of tensor dimensions, and K′ represents the flattened key tensor.

[0030] X3, calculate the original attention map: R = Q′K ′T ∈R B×(W×H)×(W×H) ;

[0031] Where T represents transpose;

[0032] X4, attention map regularization: R′=avg_pool(R)∈R B×1×W×H ;

[0033] Among them, R′ represents regularization, avg_pool(·) represents the average pooling function;

[0034] X5, normalized: attention_R = σ(R′)∈R B×1×W×H ;

[0035] Among them, attention_R represents the attention weight map after Sigmoid activation, and σ(·) represents Sigmoid activation;

[0036] X6, the final output feature generation: Ouput = V⊙attention_R∈R B×C×W×H ;

[0037] Among them, Output represents the output features after refinement processing, and ⊙ represents element-by-element multiplication.

[0038] As a preferred technical solution, a global context relationship module is used to generate a global context modeling feature G, and then G is decoded by a decoder to generate a segmentation map.

[0039] As a preferred technical solution, both MobileNetV3 networks are of N level, where N≥2 and N is an integer.

[0040] As a preferred technical solution, the semantic guidance feature CRc is generated by channel connection aggregation features, and then relational modeling is used to represent the relationship between channels. The formula is as follows:

[0041] A = [Conv2D(CRc)] D *Conv2D(CRc)

[0042]

[0043] Among them, A represents the relationship matrix, Conv2D(·) represents two-dimensional convolution, D represents, CRc represents the extraction of its spatial features, RM represents the relationship modeling matrix, Conv1D(·) represents one-dimensional convolution, β(·) represents the nonlinear activation function, exp(·) represents the exponential function, c represents the number of channels, i represents the row index, and j represents the column index.

[0044] As a preferred technical solution, N=5.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] (1) The present invention introduces dual-light fusion technology for fusion infrared images, that is, combining the color information of traditional RGB images with infrared data, which can significantly improve the model's understanding of the three-dimensional structure of the scene and enhance its adaptability to changes in illumination and texture. This fusion of technology not only improves the accuracy of leakage detection, but also optimizes data processing efficiency, providing more reliable and accurate technical support for health monitoring of concrete structures.

[0047] (2) The deep learning model of the present invention using dual-light fusion technology that fuses infrared images can provide higher segmentation quality and model robustness when processing concrete structure images with complex lighting and textures. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A schematic diagram of the architecture of a leakage analysis method based on multimodal image fusion described in the present invention. DETAILED DESCRIPTION

[0049] The present invention will be further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0050] Example 1

[0051] like Figure 1 As shown in the figure, the present invention proposes a concrete leakage segmentation network that integrates infrared images. This network combines RGB images and infrared data to achieve a segmentation method that has a deeper understanding of concrete leakage characteristics. Infrared information provides the model with additional spatial dimension information, which enables it to have stronger discrimination ability in dealing with shadows, reflections and intrinsic textures of materials. In addition, the addition of infrared data significantly improves the model's positioning accuracy of leakage edges, especially when the leakage morphology is irregular or highly similar to the concrete surface texture.

[0052] 1.1 Architecture Overview

[0053] The improvements proposed in this paper are mainly focused on improving the multimodal data processing capabilities and the accuracy of the feature fusion enhancement module. This paper uses two MobileNetV3 networks as the basis for feature extraction, processing RGB images and infrared images respectively. We process the multi-level RGB and infrared image features of the i-th block (i∈{1,2,3,4,5}) of MobileNet-V3, representing Figure 1 Zhongwei and These features are then processed by the decoder layer and independently supervised to generate a coarse segmentation map S rgb and S depth In order to establish the dependency relationship between RGB and infrared image features and achieve multimodal complementarity, we perform and ). In addition, the GCRM ​​module is introduced to explore global context clues and provide global guidance for feature interactions. A decoder is connected to the GCRM ​​module to generate a global coarse segmentation map S global . In order to further improve the segmentation performance, the ARM module is used to aggregate detailed features. We use residual learning to design an upstream in which auxiliary supervision and target supervision run in parallel. The upstream consists of four stages, in which auxiliary supervision is carried out in sequence, including attention supervision, binary supervision, and boundary supervision. The target supervision only focuses on semantic segmentation supervision. Finally, through gradual refinement, we obtain an accurate semantic segmentation map. The segmentation map P4 predicted by the fourth stage is taken as the final segmentation result.

[0054] 1.2 Global Context Module

[0055] In this study, we introduced the Global Context Relationship Module (GCRM), which is designed to optimize the feature representation of seepage in plunge pool corridors. The GCRM ​​module can perform in-depth analysis and refinement on any given intermediate feature graph by comprehensively considering the feature information of both nodes and channels.

[0056] The leakage features in the cushion pond corridor are usually small and account for a small proportion of the image, which is easily affected by complex environmental factors. In order to improve the distinction between leakage features and background, this module further intervenes after the backbone feature extraction network extracts preliminary features. By fusing the information of each channel and strengthening the feature response of specific locations, GCRM ​​significantly improves the recognition efficiency and accuracy of leakage features. In addition, this module realizes adaptive feature optimization by element-by-element multiplication with the input feature map, further enhancing the model's ability to capture the details of leakage in the cushion pond corridor, enabling it to work effectively in complex environments.

[0057] The GCRM ​​module is used to generate a global context modeling feature G, which will serve as a global guide and assist supervision with residual learning. Both auxiliary tasks and target tasks require global context information to provide target location clues. The key is to improve the semantic representation of each channel through global context modeling.

[0058] To solve this problem, Figure 1 As shown in Figure 1, we first aggregate three high-level complementary features CR3, CR4, and CR5 through channel connections to generate the semantically guided feature CRc. Next, we use relational modeling (RM) to represent the relationship between channels as follows:

[0059] A = [Conv2D(CRc)] D *Conv2D(CRc) (1)

[0060]

[0061] Through this approach, we are able to significantly improve the performance of the model in detecting seepage in plunge pond galleries, enabling it to effectively identify leakage characteristics even in complex environments.

[0062] 1.3 Feature Fusion Enhancement Module (FFEM)

[0063] The core challenge of multimodal information is to effectively mine the complementary relationships between different semantic regions. To this end, we introduce a feature fusion enhancement module that aims to construct an interaction space to explain the long-range complementary relationships between these regions in spatial and channel dimensions. This module treats the input data as a series of features (pixels) that contain different feature dimensions such as spatial structure, color, infrared image, and local and global texture. In this way, the module is able to promote information exchange between contexts and enhance the performance of multimodal features at the local and global levels.

[0064] In the water cushion pond corridor leakage detection scenario, the leakage features are usually small and account for a low proportion in the image, and are easily affected by complex environmental factors. In order to improve the distinction between leakage features and background, the feature fusion enhancement module we introduced significantly improves the recognition efficiency and accuracy of leakage features by fusing the information of each channel and strengthening the feature response of specific locations. In addition, this module realizes adaptive feature optimization by element-by-element multiplication with the input feature map, further enhancing the model's ability to capture the details of water cushion pond corridor leakage, enabling it to work effectively in complex environments.

[0065] The feature fusion enhancement module not only generates global context modeling features G, but also combines auxiliary supervision through residual learning. Both auxiliary tasks and target tasks require global context information to provide target location clues. The key is to improve the semantic representation of each channel through global context modeling, thereby improving the performance of the model in water cushion pond corridor leakage detection.

[0066] In the task of detecting leakage in the cushion pond corridor, the introduction of Multihead Attention greatly improves the model's ability to process complex images and identify subtle features. Traditional convolutional networks often rely on multi-layer stacking to increase the receptive field, while the multi-head self-attention mechanism allows the model to directly establish dependencies on a global scale. This means that the network can understand and integrate features that are far away within a single layer, which is particularly important for accurate detection of leakage areas.

[0067] First, the input is linearly transformed. Each input (query A, key B, value C) is first transformed independently to prepare a specific representation for each head:

[0068]

[0069] In the scenario of leakage detection in cushion pond corridors, leakage features are usually small and account for a small proportion of the image, and are easily affected by complex environmental factors. The multi-head self-attention mechanism significantly improves the recognition efficiency and accuracy of leakage features by capturing long-range feature dependencies in a global scope. In addition, the mechanism combines the results of multiple self-attention heads to enhance the model's ability to capture the details of leakage in cushion pond corridors, enabling it to work effectively in complex environments.

[0070] In this way, the model can generate more accurate and stable feature representations, significantly improving the performance of the cushion pond gallery leakage detection task.

[0071] in, is a weight matrix that transforms the input to the new space of the corresponding head. These transformations are trainable and are intended to extract different features of the input data.

[0072] Use Softmax to calculate the normalized attention weights: This process reveals the strength of the association between features: This ensures that the sum of all weights is 1.

[0073] where d m is the dimension of B, where the dot product is divided by This is to avoid the gradient vanishing problem caused by too large values ​​in the softmax operation.

[0074] Finally, the outputs of all heads are combined and passed through another linear transformation U O Get the final output: MultiHead(A,B,C)=Concat(head1,…,head h ) O

[0075] Among them, the output head of each head i It is an independently calculated Attention (A i ,B i ,C i ).

[0076] 1.4 Attention Refinement Module (ARM) and Cross Attention Module (CAM)

[0077] In this study, the attention refinement module combines the RGB and infrared image features F i R and F i T(i=1,2) are connected. In order to significantly reduce the computational cost while maintaining performance, we use depthwise separable convolution to obtain comprehensive features. In order to extract deep features, we use global maximum pooling and convolution to enhance the extraction of spatial information. Subsequently, we obtain detailed aggregation features through the cross-attention module. In the attention refinement module, we introduce a new cross-attention module, which is specially designed to enhance the expression ability of the seepage features of the cushion pond corridor in complex backgrounds. Through detailed feature fusion strategies and cross-attention mechanisms, this module optimizes the intermediate feature maps and improves the performance and adaptability of the model in changing environments. In the task of cushion pond corridor seepage detection, the leakage features are usually small and account for a small proportion in the image, and are easily affected by complex environmental factors. The cross-attention module significantly improves the recognition efficiency and accuracy of the leakage features by establishing associations between different feature maps. By combining the information of multiple feature maps, the module can more accurately capture the details of the leakage area in complex backgrounds.

[0078] Specifically, the cross-attention module first refines the input features and enhances the feature expression in the spatial and channel dimensions through the cross-attention mechanism. Then, through the feature fusion strategy, the feature information from different sources is integrated to generate an optimized intermediate feature map. This process significantly improves the adaptability and performance of the model in complex environments, enabling it to work effectively in the variable water cushion pond corridor leakage detection scenario.

[0079] Given an input feature X with a dimension of R B×C×W×H Where: B is the batch size, C is the number of channels, W and H are the width and height of the feature map respectively. Then perform feature conversion and Key conversion: This operation reduces the dimension through a 1×1 convolution layer to reduce the complexity of subsequent calculations. The same 1×1 convolution layer is used for dimensionality reduction. Value conversion: V = value_conv(X)∈R B×C×W×H Here V retains the original number of channels for multiplication with the attention map. Then the attention weights are calculated and rearranged:

[0080]

[0081]

[0082] Calculate the original attention map: R = Q′K ′T ∈R B×(W×H)×(W×H) This step calculates the correlation strength between all spatial positions through matrix multiplication. Then the attention map is regularized and average pooled: R′=avg_pool(R)∈RB×1×W×H Adaptive average pooling is used here to reduce the spatial dimension in each batch, thereby obtaining the average attention at each position. Sigmoid activation: attention_R = σ(R′)∈R B×1×W×H Use the Sigmoid function for normalization and limit the attention value to the range of [0,1]. Finally, the output feature is generated: Output = V⊙attention_R∈R B×C×W×H The final output is the element-wise product of the original value feature and the normalized attention map attention_R. This ensures that more attention is paid to useful features when features are fused.

[0083] Through this fine-grained attention adjustment, the cross-attention module significantly improves the discrimination of leakage features in complex backgrounds, helping to more accurately identify and analyze the location, morphology, and distribution of leakage areas. Leakage areas in the plunge pool corridor may vary in size, ranging from small-scale subtle leakage to large-scale significant leakage. The module's adaptive feature optimization capability enables it to maintain high performance in different visual scenes, especially when the leakage features are similar to background noise. This ability is particularly important.

[0084] In the task of detecting leakage in the pool corridor, leakage features are often affected by complex environmental factors. The cross-attention module significantly improves the recognition efficiency and accuracy of leakage features by establishing associations between different feature maps. By combining the information of multiple feature maps, the module can more accurately capture the details of the leakage area in a complex background.

[0085] Specifically, the cross-attention module first refines the input features and enhances the feature expression in the spatial and channel dimensions through the cross-attention mechanism. Then, through the feature fusion strategy, the feature information from different sources is integrated to generate an optimized intermediate feature map. This process significantly improves the adaptability and performance of the model in complex environments, enabling it to work effectively in the variable water cushion pond corridor leakage detection scenario, regardless of the size of the leakage area.

[0086] As described above, the present invention can be preferably implemented.

[0087] All features disclosed in all embodiments in this specification, or steps in all methods or processes implicitly disclosed, except for mutually exclusive features and / or steps, can be combined and / or expanded or replaced in any manner.

[0088] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. According to the technical essence of the present invention, within the spirit and principles of the present invention, any simple modification, equivalent replacement and improvement made to the above embodiment still falls within the protection scope of the technical solution of the present invention.

Claims

1. A leakage analysis method based on multimodal image fusion, characterized in that: A MobileNetV3 network is used to extract features from RGB images to obtain RGB features, and another MobileNetV3 network is used to extract features from infrared images to obtain infrared image features; then the RGB features and infrared image features are decoded through a decoder to generate a segmentation map.

2. The leakage analysis method based on multimodal image fusion according to claim 1, characterized in that: The feature fusion and enhancement module is used to fuse and enhance the RGB features and infrared image features.

3. The leakage analysis method based on multimodal image fusion according to claim 1, characterized in that: The multi-head self-attention mechanism is used to process RGB features and infrared image features, including the following steps: D1, linearly transforms each input to prepare representation for each head, the formula is: In the formula, A i represents the query after linear transformation, and A represents the query before linear transformation. represents the query weight matrix, B i represents the key after linear transformation, B represents the key before linear transformation, represents the key weight matrix, C i represents the value after linear transformation, C represents the value before linear transformation, represents the value weight matrix; D2 is calculated and scaled by the attention weights, and the formula is: In the formula, Attention(A i ,B i ,C i ) represents the focus filter, d m represents the dimension of B; D3 combines the outputs of all heads and obtains the final output through linear transformation. The formula is: MultiHead(A,B,C)=Concat(head1,…,head h )U O In the formula, MultiHead(A,B,C) represents the final output, Concat(head1,…,head h ) represents the output of all heads after merging, U O represents the linear transformation matrix, h represents the number of heads; Among them, the output head of each head i For the independently calculated Attention (A i ,B i ,C i ).

4. The leakage analysis method based on multimodal image fusion according to claim 1, characterized in that: The ARM module is used to aggregate RGB features and infrared image features.

5. The leakage analysis method based on multimodal image fusion according to claim 1, characterized in that: The cross-attention module is used to process the RGB features and infrared image features: first, the features are refined, and the feature expression is enhanced in the spatial dimension and channel dimension through the cross-attention mechanism; then, the input features from different sources are fused through the feature fusion strategy to generate an optimized intermediate feature map.

6. The leakage analysis method based on multimodal image fusion according to claim 5, characterized in that: Refine the features The following steps are involved: X1, with dimension R B×C×W×H The input feature X is transformed; Among them, B is the batch size, C is the number of channels, W is the width of the feature map, H is the height of the feature map, and r is the reduction ratio, including the following transformations: Key conversion: Query conversion: Value conversion: V = value_conv(X)∈R B×C×W×H ; Where K represents the key, key_conv(·) represents the key conversion function, R represents the attention map, Q represents the query, query_conv(·) represents the query conversion function, V represents the value, and value_conv(·) represents the value conversion function; X2, calculate the attention weights and rearrange them: Where Q′ represents the query tensor after flattening and transposition, view(·) represents reshaping the tensor, permute(·) represents changing the order of tensor dimensions, and K′ represents the flattened key tensor. X3, calculate the original attention map: R = Q′K ′T ∈R B×(W×H)×(W×H) ; Where T represents transpose; X4, attention map regularization: R′=avg_pool(R)∈R B×1×W×H ; Among them, R′ represents regularization, avg_pool(·) represents the average pooling function; X5, normalized: attention_R = σ(R′)∈R B×1×W×H ; Among them, attention_R represents the attention weight map after Sigmoid activation, and σ(·) represents Sigmoid activation; X6, the final output feature generation: Output = V⊙attention_R∈R B×C×W×H ; Among them, Output represents the output features after refinement processing, and ⊙ represents element-by-element multiplication.

7. The leakage analysis method based on multimodal image fusion according to claim 1, characterized in that: The global context relationship module is used to generate the global context modeling feature G, and then G is decoded by the decoder to generate a segmentation map.

8. The leakage analysis method based on multimodal image fusion according to claim 1, characterized in that: Both MobileNetV3 networks are of N levels; where N ≥ 2 and N is an integer.

9. The leakage analysis method based on multimodal image fusion according to claim 8, characterized in that: The semantic guided feature CRc is generated by channel connection aggregation features, and then relational modeling is used to represent the relationship between channels. The formula is as follows: A=[Conv2D(CRc)] D *Conv2D(CRc) Among them, A represents the relationship matrix, Conv2D(·) represents two-dimensional convolution, D represents, CRc represents the extraction of its spatial features, RM represents the relationship modeling matrix, Conv1D(·) represents one-dimensional convolution, β(·) represents the nonlinear activation function, exp(·) represents the exponential function, c represents the number of channels, i represents the row index, and j represents the column index.

10. A leakage analysis method based on multimodal image fusion according to claim 8 or 9, characterized in that: N＝5。

Images