Multi-modal image fusion method based on deep coding and decoding axis interactive attention network

Abstract

The invention relates to the technical field of image processing, in particular to a multi-modal image fusion method based on a deep coding and decoding axis interactive attention network, which comprises the following steps of: S1, acquiring data of an infrared image and a visible light image, normalizing the infrared image and the visible light image, and then inputting a model for feature extraction; the feature extraction method comprises an encoder, a fusion strategy and a decoder. S2, in a feature extraction stage of an encoder, multiple times of extraction is performed on two paths of infrared and visible light images through a convolutional neural network and transformer, so that local information of two modal images is captured, and encoding representation is obtained; s3, performing multi-time layered fusion on the features coded by the encoder by a fusion strategy; s4, reconstructing and decoding the fused image by a decoder; the feature representation capability is higher, the utilization rate of original information is higher, image detail mining is more sufficient, and the fusion effect is better.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a multimodal image fusion method based on a deep encoder-decoder axis-interactive attention network. Background Technology

[0002] During image acquisition, single-modal images acquired by a single sensor cannot accurately represent comprehensive scene information. With the development of multi-sensor technology, multi-modal image fusion has a better prospect for development. Multi-modal image fusion is the process of aggregating image information from different sensors, which can improve the perception and understanding of the scene and obtain a more comprehensive image representation.

[0003] However, current multimodal image fusion methods face many challenges. First, achieving deep fusion of information from multiple modalities is the core issue at present. Different modalities exhibit significant differences in data distribution, feature dimensions, and semantic representation. Traditional fusion methods often remain at the level of shallow feature stitching or simple weighting, making it difficult to establish deep connections and semantic-level interactions between modalities, thus hindering the fusion effect from breaking through bottlenecks. Second, for practical applications, lightweight models and multi-domain applicability are the goals pursued by multimodal fusion. Existing deep learning-based fusion models often rely on complex network structures and a large number of parameters. While these can improve fusion performance, they suffer from high computational costs and slow inference speeds, making them difficult to deploy on resource-constrained terminal devices. Furthermore, most models are designed for specific scenarios (such as remote sensing or medical applications) and lack good generalization capabilities, leading to significant performance degradation when transferring across domains. Faced with the challenges of complex scenarios, existing models lack an adaptive modality weight adjustment mechanism, which can easily lead to distortion or information loss in the fusion results. In complex environments, the reliability of information in each modality can fluctuate dynamically, making it difficult to guarantee the stability and robustness of the fusion process. How to achieve effective and stable modality fusion has become a research focus. Therefore, a multimodal image fusion method based on deep encoding and decoding axis interaction attention network is urgently needed to improve the above problems. Summary of the Invention

[0004] To address the aforementioned technical challenges, this invention provides a multimodal image fusion method based on a deep encoder-decoder axis interaction attention network. This method enhances feature representation capabilities by expanding the receptive field through dilated convolution and strengthening global-local feature interaction through a Transformer structure and skip connections. It captures richer contextual information, effectively associates modal features at different levels, and improves the completeness and accuracy of feature representation. Furthermore, it achieves higher utilization of original information through efficient fusion strategies and multimodal attention extraction strategies. This fully aggregates features from different attention dimensions, strengthens hidden representations within modal features, and maximizes the preservation and utilization of original multimodal image information. It also provides more comprehensive image detail mining through a progressive fusion method at the encoding layer, enabling feature interaction between different depth levels. This allows for more comprehensive mining of diverse texture details in the image, reducing the loss of detail information and improving the texture clarity and detail richness of the fused image. Finally, it delivers superior fusion results through optimization of the entire process from structural design and feature interaction to information transmission. This results in improved multimodal image fusion performance, enhancing modal complementarity and visual quality, and making it more suitable for subsequent image analysis and application needs.

[0005] The multimodal image fusion method based on a deep encoder-decoder axis-interaction attention network of the present invention includes the following steps: S1. Acquire infrared and visible light image data, normalize the infrared and visible light images, and then input them into the model for feature extraction. The feature extraction method includes an encoder, a fusion strategy, and a decoder; S2. The encoder performs feature extraction in the following stage: it extracts infrared and visible light images multiple times through convolutional neural networks and transformers to capture local information of the two modal images and obtain coded representations. S3. The fusion strategy performs multiple layered fusions on the features encoded by the encoder. Through axis interaction attention, the encoded information of the two modes can be deeply fused. Improved interaction attention enables the features of different modes to flow and transfer, resulting in richer cross-modal fusion information. S4. The decoder reconstructs and decodes the fused image, gradually restoring the decoded feature information into a fused image with features of both modalities.

[0006] Preferably, the detailed steps of the encoder's feature extraction stage in S2 include: S2-1. Input infrared and visible light images, and extract the corresponding dual-path coding blocks 1, 2, and 3. That is, extract features from infrared image coding blocks 1, 2, and 3, as well as visible light image coding blocks 1, 2, and 3. Dual-path coding blocks 1, 2, and 3 have the same structure. For dual-path coding block 1, a 3x3 separable convolutional layer is first used for preliminary extraction, and the ReLU activation function is introduced to increase its nonlinearity. S2-2. After that, feature encoding is performed through three coding layers with the same structure. Each coding layer consists of two deep convolutional blocks. Each deep convolutional block contains a dilated convolutional end connected to a separable convolution. The dilated convolution uses a 3x3 kernel, a dilation rate of 2, and padding of 2. The separable convolution uses a 1x1 kernel. A ReLU activation function is introduced after each convolution to increase non-linearity. S2-3 Finally, after feature extraction through three encoding layers, the obtained feature information is fed into the Transformer layer. Its structure begins with a window function dividing the features into blocks. After flattening the blocks, feature extraction is performed using a multi-head attention mechanism, and finally, the query vector is obtained through three 3x3 convolutions. Key vector Value vector , , The multi-head attention weights are obtained after softmax weight calculation. Matrix transpose and transpose operations are performed to obtain multi-head attention features. Then, normalized features are obtained through layer normalization. The normalized features are aggregated with the encoded features before being fed into the multi-head attention and subjected to MLP operation. After normalization, Transformer attention features are obtained, which are the features extracted by dual-path coding block 1. For the feature images with infrared and visible light dual-path input, the extraction methods of dual-path coding blocks 2 and 3 are the same as those of dual-path coding block 1. After extraction by three dual-path coding blocks, the encoder features are finally obtained.

[0007] Preferably, the corresponding features of the obtained dual-path coding block 1, dual-path coding block 2, and dual-path coding block 3 are respectively used as two inputs of the fusion strategy for subsequent fusion extraction. The fusion strategy obtains image features of two modalities through an axis attention module, a separable channel attention module, a modal interaction attention module, and a convolution correction module, and calculates the feature importance of each position through multiple attention modules to obtain the weight information of the two modal features. The attention of the image region of interest information is dynamically adjusted to achieve efficient modal fusion.

[0008] Preferably, the shaft attention module: To quickly extract global spatial feature information, axis attention layers are applied on both the width and height axes. The axis attention employs a multi-head attention mechanism. The calculation process for high-axis attention is as follows:

[0009] in, Represents the input matrix, dimension , It is the length of the input matrix. It is the width of the input matrix. The dimension is , The dimension is , The dimension is , , , These represent the extracted corresponding features, with dimensions of respectively. , , ; Calculate the attention along the high axis, with attention weight scores for all rows in each column:

[0010] Softmax represents the normalization function. This represents the high-axis attention feature of the output.

[0011] For wide-axis attention, the specific process is as follows:

[0012] in, Represents the input matrix, dimension Compared to the high axis, the wide axis swaps the rows and columns of the input matrix. It is the length of the input matrix. It is the width of the input matrix. , , Represents a query, key, and value matrix. The dimension is , The dimension is , The dimension is , , , These represent the extracted corresponding features, with dimensions of respectively. , , ; Calculate the attention along the wide axis, with attention weight scores for all rows in each column:

[0013] Softmax represents the normalization function. This represents the wide-axis attention feature of the output.

[0014] By concatenating wide-axis attention and high-axis attention, attention is calculated separately in the width and height directions to extract global attention. Compared to non-local self-attention, this method of calculating attention separately for the width and height axes is more efficient and faster. Multimodal image features, after undergoing axis attention, acquire more spatially expressive features. , To obtain better channel dimension features, the features extracted by axis attention are transformed into separable channel attention modules after dimension-corrected convolution. The main operations of these modules include separable convolution, global pooling, fully connected operations, normalization, and dimension restoration.

[0015] Preferably, the separable channel attention module: Image feature dimensions are compressed using separable convolution and global pooling operations, concentrating spatial feature information into channels and aggregating it into a single vector. This vector is then incorporated into a fully connected layer for feature full connectivity, while an activation function is applied to increase its non-linearity. Finally, softmax is used for normalization to ensure that the sum of the calculated weight components is 1. For the input features... The calculation process for the separable channel attention module is as follows:

[0016] in, This indicates a separable operation, using a convolution with a kernel of 1. Indicates global average pooling. This indicates a fully connected operation. It's the activation function; here, ReLU is used. This represents the Softmax activation function. This represents the corresponding attention weight for capture, which ranges from [0,1]. The features extracted by the separable channel attention are then passed to the modal interaction attention module.

[0017] Preferably, the modal interaction attention module: Multi-head interactive attention is used to model long-distance dependencies between different modalities. For features extracted by the encoder, ( ), This indicates the number of layers in the encoder that extract features; it consists of four layers. Used to distinguish modes , This represents a visible and infrared multimodal image, with a size of [size missing]. , Indicates the feature map height. Indicates the width of the feature map. This represents the number of feature map channels, and the initial extraction is performed using a size of [size missing]. Local window, This indicates the length and width of a local window. Reintegrated into , This represents the dimension of the feature within each local window. The number of windows is determined by the total number of pixels in the feature map. Divide by the number of pixels in a single window After obtaining the result, a self-attention extraction operation is performed on each local window, and a weight learning matrix is ​​then used. For local window features Perform mapping. Represents the local window feature matrix. Represents the real number field. It is the dimension of the matrix, that is, we have OK, The real matrix of columns, to obtain the query ,key ,value , subscript This represents a visible light diagram, with subscripts indicating the direction of light. Infrared image representation:

[0018]

[0019] Then calculate and another mode The dot product is then normalized using the Softmax function to obtain the attention weight values. For each attention head, the calculation process is as follows:

[0020] in, Indicator key Dimensions Is a key Dimension The square root serves to avoid dimensionality. An excessively large value causes the gradient of the softmax function to vanish. Indicates query Perform a dot product operation on the transpose of the bond to obtain a matrix representation. and Similarity score, This represents the output of a single attention head. The obtained self-attention is then extended to multi-head interactive self-attention to extract different attention distributions and capture information from different dimensions. Due to the above extraction , and The information comes from two modalities, and the cross-modal extraction process can be represented as follows:

[0021]

[0022] Among them, the visible light mode With infrared mode and Perform cross-modal information aggregation. This indicates multi-head interactive attention. The representation layer is normalized, and residual connections are introduced to retain the initial information of the visible light modes in order to establish the connection and interaction between different modes and keep the input information from being lost.

[0023]

[0024] in, and This indicates the intermediate features of the multi-head interactive attention mechanism extracted from visible light and infrared light. Indicates a feedforward network. Representation layer normalization, and This represents the extracted multi-head interaction attention features; at last, and Channel concatenation is performed, and dimensionality correction is applied using convolution to obtain fused features:

[0025] in This indicates the fusion feature.

[0026] Preferably, the decoder consists of decoding block 1, decoding block 2 and decoding block 3. The decoder has the same structure as the encoder's convolutional layer and encoding layer, but omits the Transformer feature extraction part and introduces cross-layer skip connections and residual networks to make full use of the feature decoding of each layer. Finally, the decoder gradually extracts and obtains the fused image.

[0027] Preferably, after the image fusion is completed, a combination of pixel loss, structural similarity loss, and gradient loss is used for loss calculation. The loss function applies pixel loss, structural similarity loss, and gradient loss, and the calculation process of the loss function is as follows:

[0028] in, Indicates total training loss. Indicates pixel loss, Represents structural similarity loss. For gradient loss; Pixel loss is used to constrain the similarity in pixel values ​​between the fused image and the source image, and its calculation can be expressed as:

[0029] in, The corresponding pixel values ​​of the input image. To output the pixel values ​​of the image, For the position of the corresponding pixel, Indicates the number of iterations; Structural similarity loss is used to measure the structural similarity between the fused image and the source image, and its calculation can be expressed as:

[0030]

[0031] in, It is the input image. It outputs an image. It is a structural similarity function. , The mean of the input and output images. , A standard table for input and output images. For the input image and output image covariance, , , This is a constant term set to avoid a denominator of 0, where This is a constant term set to avoid a denominator of 0. , , ; Gradient loss is used to preserve edge and texture details of an image, and its calculation can be expressed as:

[0032] in, It is an image gradient, It is an image The gradient.

[0033] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. Enhanced feature representation capabilities: By expanding the receptive field of features through dilated convolution and combining the Transformer structure with skip connections to strengthen global-local feature interaction, it can capture richer contextual information, effectively associate modal features at different levels, and improve the completeness and accuracy of feature representation. 2. Higher utilization of original information: The efficient fusion strategy and multimodal attention extraction strategy can fully aggregate features from different attention dimensions, strengthen the hidden representation in modal features, and retain and utilize the original multimodal image information to the maximum extent. 3. More thorough image detail mining: The gradual transmission and fusion method of the coding layer enables feature interaction between different depth levels, which can more comprehensively mine diverse texture details of the image, reduce the loss of detail information, and improve the texture clarity and detail richness of the fused image. 4. Superior fusion effect: The entire process of optimization, from structural design (encoding and decoding reconstruction), feature interaction (multi-attention combination) to information transmission (layered fusion), ultimately achieves a better multimodal image fusion effect, which improves the modal complementarity and visual quality of the fused image, making it more suitable for subsequent image analysis and application needs. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the encoder block structure of each branch of the present invention; Figure 2 This is a schematic diagram of the decoder block structure of the present invention; Figure 3 This is a schematic diagram of the modal axis interaction attention fusion strategy structure of the present invention; Figure 4 This is a schematic diagram of the multimodal image fusion method using a deep encoding / decoding axis interaction attention network according to the present invention. Detailed Implementation

[0035] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. The present invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0036] Example: Figures 1 to 4As shown, the multimodal image fusion method based on deep encoding and decoding axis interaction attention network focuses on the fusion of infrared and visible light images. Infrared images utilize the principle of thermal radiation and have the characteristics of strong penetration and gradient information sensitivity. Visible light images are acquired using the principle of reflected light and have the characteristics of high spatial resolution and sufficient preservation of texture details. By employing an autoencoder network framework, the training process involves an encoder mapping the input feature image into a multi-layered representation, followed by a decoder restoring the features to a representation close to the input image. Through end-to-end learning between the encoder and decoder, the mapping is learned directly from the input and output data, eliminating the tedious process of manually designing extraction methods. After model training, the encoder, fusion strategy, and decoder jointly achieve multimodal image fusion. The encoder extracts features from visible light and infrared images; the fusion strategy fuses the features extracted from the two sources; and the decoder reconstructs the features from the fused image to obtain the multimodal fused image. The main encoder part adopts a hybrid architecture combining CNN (Convolutional Neural Network) and Transformer. For the different characteristics of visible light and infrared images, a dual-branch encoder structure is used. Each branch contains one convolutional layer, three coding layers, and one Transformer feature extraction layer. The convolutional layers use a size of... Separable convolutions, each coding layer consists of two dilated convolutions and two depthwise separable convolutions. The dilated convolutions use a kernel size of [missing value]. The dilation rate is set to 2, the padding is set to 2, and the depthwise separable convolution uses a kernel size of [missing value]. The Transformer features a convolutional layer, with each convolution followed by an activation function layer using ReLU. The feature extraction layer comprises a multi-head attention layer, an MLP layer, and two normalization layers. Its structure begins with a window function dividing the features into blocks. These blocks are then flattened and extracted using a multi-head attention mechanism. Finally, three 3x3 convolutions are used to process the data and obtain the query vector. Key vector Value vector , and The multi-head attention weights are obtained after softmax weight calculation. Matrix transpose and transpose operations are performed to obtain multi-head attention features. Then, layer normalization is used to obtain normalized features. The normalized features are aggregated with the encoded features before being fed into the multi-head attention layer and subjected to MLP operation. After normalization, Transformer attention features are obtained. The above extraction methods are applied to visible light and infrared image features to extract multimodal layered features. Then, the layered extraction of different modalities is fed into the fusion strategy to integrate multimodal information and obtain fused features. Finally, the obtained fused features are fed into the decoder. The decoder is designed with a similar structure to the encoder, but the Transformer feature extraction layer is removed, and cross-layer skip connections and residual networks are introduced. The purpose of this is to make full use of the feature decoding of each layer. Finally, the fused image is obtained by gradually extracting features through the decoder. Most current fusion methods employ convolutional neural networks to aggregate multimodal information. However, convolutional operations, due to their limited receptive field, struggle to capture long-range dependencies, resulting in poor fusion performance when fusing complex scene images. Furthermore, considering that multimodal image sources originate from different sensors, modal interaction should be maximized during fusion to extract more comprehensive cross-modal image information. Therefore, this invention designs a modal interaction axis-attention fusion strategy based on the Transformer architecture to overcome the poor fusion results caused by low-quality or highly complex multimodal images. The modal axis-attention fusion strategy consists of an axis-attention module, a separable channel attention module, a modal interaction attention module, and a convolution correction module, as follows... Figure 3 As shown; The axis attention module is used to quickly extract global spatial feature information. Axis attention layers are applied on both the width and height axes. The axis attention uses a multi-head attention mechanism. The calculation process for high-axis attention is as follows:

[0037] in, Represents the input matrix, dimension , It is the length of the input matrix. It is the width of the input matrix. The dimension is Used to extract column features (dimensions) from the input matrix. ) converted into query features (dimensions) ), The dimension is Used to extract column features (dimensions) from the input matrix. ) converted into key features (dimensions) ), The dimension is Used to extract column features (dimensions) from the input matrix. Converting ) to value features (dimensions) ), Indicates query ( ) and key ( The feature dimension of the matrix, i.e., the vector length of the query and key. Represents value ( The eigendimensional dimension of a matrix, i.e., the vector length of its values. , , These represent the extracted corresponding features, with dimensions of respectively. , , ; Calculate the attention along the high axis, with attention weight scores for all rows in each column:

[0038] Softmax represents the normalization function. This represents the axis attention feature of the output.

[0039] For wide-axis attention, the specific process is as follows:

[0040] in, Represents the input matrix, dimension Compared to the high axis, the wide axis swaps the rows and columns of the input matrix. It is the length of the input matrix. It is the width of the input matrix. , , Represents a query, key, and value matrix. The dimension is Used to extract column features (dimensions) from the input matrix. ) converted into query features (dimensions) ), The dimension is Used to extract column features (dimensions) from the input matrix. ) converted into key features (dimensions) ), The dimension is Used to extract column features (dimensions) from the input matrix. Converting ) to value features (dimensions) ), Indicates query ( ) and key ( The feature dimension of the matrix, i.e., the vector length of the query and key. Represents value ( The eigendimensional dimension of a matrix, i.e., the vector length of its values. , , These represent the extracted corresponding features, with dimensions of respectively. , , ; Calculate the attention along the wide axis, with attention weight scores for all rows in each column:

[0041] By concatenating wide-axis attention and high-axis attention, attention is calculated separately in the width and height directions to extract global attention. Compared to non-local self-attention, this method of calculating attention separately for the width and height axes is more efficient and faster. Multimodal image features, after undergoing axis attention, acquire more spatially expressive features. , In order to obtain better channel dimension features, the features extracted by axis attention are converted into separable channel attention modules after dimension-corrected convolution. The main operations include separable convolution, global pooling, fully connected operation, normalization, and dimension restoration. The separable channel attention module compresses image feature dimensions through separable convolution and global pooling operations, compressing spatial feature information into channels and aggregating it into a single vector. This vector is then incorporated into a fully connected layer for feature full connectivity, while an activation function is applied to increase its non-linearity. Finally, softmax is used for normalization to ensure that the sum of the calculated weight components is 1. For the input features... The calculation process for the separation channel attention module is as follows:

[0042] in, This indicates a separable operation, using a convolution with a kernel of 1. Indicates global average pooling. This indicates a fully connected operation. It's the activation function; here, ReLU is used. This represents the Softmax activation function. This represents the corresponding attention weight captured, which ranges from [0,1]. The features extracted by the separable channel attention are then passed to the modal interaction attention module. The modal interaction attention module models long-distance dependencies between different modalities through multi-head interaction attention. For features extracted by the encoder, ( ), This indicates the number of layers in the encoder that extract features; it consists of four layers. Used to distinguish modes , This represents a visible and infrared multimodal image, with a size of [size missing]. , Indicates the feature map height. Indicates the width of the feature map. This represents the number of feature map channels, and the initial extraction is performed using a size of [size missing]. Local window, This indicates the length and width of a local window. Reintegrated into , This represents the dimension of the feature within each local window. The number of windows is determined by the total number of pixels in the feature map. Divide by the number of pixels in a single window After obtaining the result, a self-attention extraction operation is performed on each local window, and a weight learning matrix is ​​then used. For local window features Perform mapping. Represents the local window feature matrix. Represents the real number field. It is the dimension of the matrix, that is, we have OK, The real matrix of columns, to obtain the query ,key ,value , subscript This represents a visible light diagram, with subscripts indicating the direction of light. Infrared image representation:

[0043]

[0044] Then calculate and another mode The dot product is then normalized using the Softmax function to obtain the attention weight values. For each attention head, the calculation process is as follows:

[0045] in, Indicator key Dimensions Is a key Dimension The square root serves to avoid dimensionality. An excessively large value causes the gradient of the softmax function to vanish. Indicates query Perform a dot product operation on the transpose of the bond to obtain a matrix representation. and Similarity score, This represents the output of a single attention head. The obtained self-attention is then extended to multi-head interactive self-attention to extract different attention distributions and capture information from different dimensions. Due to the above extraction , and The information comes from two modalities, and the cross-modal extraction process can be represented as follows:

[0046]

[0047] Among them, the visible light mode With infrared mode and Perform cross-modal information aggregation. This indicates multi-head interactive attention. The representation layer is normalized, and residual connections are introduced to retain the initial information of the visible light modes in order to establish the connection and interaction between different modes and keep the input information from being lost.

[0048]

[0049] in, and This indicates the intermediate features of the multi-head interactive attention mechanism extracted from visible light and infrared light. Indicates a feedforward network. Representation layer normalization, and This represents the extracted multi-head interaction attention features; at last, and Channel concatenation is performed, and dimensionality correction is applied using convolution to obtain fused features:

[0050] in Indicates fusion characteristics; The overall network structure diagram is as follows Figure 4 As shown.

[0051] The main functions achieved by this invention are: 1. Enhanced feature representation capabilities: By expanding the receptive field of features through dilated convolution and combining the Transformer structure with skip connections to strengthen global-local feature interaction, it can capture richer contextual information, effectively associate modal features at different levels, and improve the completeness and accuracy of feature representation. 2. Higher utilization of original information: The efficient fusion strategy and multimodal attention extraction strategy can fully aggregate features from different attention dimensions, strengthen the hidden representation in modal features, and retain and utilize the original multimodal image information to the maximum extent. 3. More thorough image detail mining: The gradual transmission and fusion method of the coding layer enables feature interaction between different depth levels, which can more comprehensively mine diverse texture details of the image, reduce the loss of detail information, and improve the texture clarity and detail richness of the fused image. 4. Superior fusion effect: The entire process of optimization, from structural design (encoding and decoding reconstruction), feature interaction (multi-attention combination) to information transmission (layered fusion), ultimately achieves a better multimodal image fusion effect, which improves the modal complementarity and visual quality of the fused image, making it more suitable for subsequent image analysis and application needs.

[0052] Technical personnel in this industry only need to follow the accompanying instruction manual for installation and operation, without requiring any creative effort from those skilled in the art.

[0053] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A multimodal image fusion method based on a deep encoder-decoder axis-interactive attention network, characterized in that, Includes the following steps: S1. Acquire infrared and visible light image data, normalize the infrared and visible light images, and then input them into the model for feature extraction. The feature extraction method includes an encoder, a fusion strategy, and a decoder; S2. The encoder performs feature extraction in the following stage: it extracts infrared and visible light images multiple times through convolutional neural networks and transformers to capture local information of the two modal images and obtain coded representations. S3. The fusion strategy performs multiple layered fusions on the features encoded by the encoder. Through axis interaction attention, the encoded information of the two modes can be deeply fused. Improved interaction attention enables the features of different modes to flow and transfer, resulting in richer cross-modal fusion information. S4. The decoder reconstructs and decodes the fused image, gradually restoring the decoded feature information into a fused image with features of both modalities.

2. The multimodal image fusion method based on a deep encoder-decoder axis-interactive attention network as described in claim 1, characterized in that, The detailed steps of the encoder's feature extraction stage in S2 include: S2-1. Input infrared and visible light images, and extract the corresponding dual-path coding blocks 1, 2, and 3. That is, extract features from infrared image coding blocks 1, 2, and 3, as well as visible light image coding blocks 1, 2, and 3. Dual-path coding blocks 1, 2, and 3 have the same structure. For dual-path coding block 1, a 3x3 separable convolutional layer is first used for preliminary extraction, and the ReLU activation function is introduced to increase its nonlinearity. S2-2. After that, feature encoding is performed through three coding layers with the same structure. Each coding layer consists of two deep convolutional blocks. Each deep convolutional block contains a dilated convolutional end connected to a separable convolution. The dilated convolution uses a 3x3 kernel, a dilation rate of 2, and padding of 2. The separable convolution uses a 1x1 kernel. A ReLU activation function is introduced after each convolution to increase non-linearity. S2-3 Finally, after feature extraction through three encoding layers, the obtained feature information is fed into the Transformer layer. Its structure begins with a window function dividing the features into blocks. After flattening the blocks, feature extraction is performed using a multi-head attention mechanism, and finally, the query vector is obtained through three 3x3 convolutions. Key vector Value vector , , The multi-head attention weights are obtained after softmax weight calculation. Matrix transpose and transpose operations are performed to obtain multi-head attention features. Then, normalized features are obtained through layer normalization. The normalized features are aggregated with the encoded features before being fed into the multi-head attention and subjected to MLP operation. After normalization, Transformer attention features are obtained, which are the features extracted by dual-path coding block 1. For the feature images with infrared and visible light dual-path input, the extraction methods of dual-path coding blocks 2 and 3 are the same as those of dual-path coding block 1. After extraction by three dual-path coding blocks, the encoder features are finally obtained.

3. The multimodal image fusion method based on a deep encoder-decoder axis-interactive attention network as described in claim 2, characterized in that, The corresponding features of the obtained dual-path coding block 1, dual-path coding block 2, and dual-path coding block 3 are used as two inputs in the fusion strategy for subsequent fusion extraction. The fusion strategy obtains image features of two modalities through an axis attention module, a separable channel attention module, a modal interaction attention module, and a convolution correction module. It also calculates the feature importance of each position through multiple attention modules to obtain the weight information of the two modal features, dynamically adjusts the attention of the image region of interest information, and achieves efficient modal fusion.

4. The multimodal image fusion method based on a deep encoder-decoder axis interaction attention network as described in claim 3, characterized in that, The axis attention module: To quickly extract global spatial feature information, axis attention layers are applied on both the width and height axes. The axis attention layers employ a multi-head attention mechanism. The calculation process for the high-axis attention is as follows: ； in, Represents the input matrix, dimension , It is the length of the input matrix. It is the width of the input matrix. The dimension is , The dimension is , The dimension is , , , These represent the extracted corresponding features, with dimensions of respectively. , , ; Calculate the attention along the high axis, with attention weight scores for all rows in each column: ； Softmax represents the normalization function. This represents the high-axis attention features of the output; For wide-axis attention, the specific process is as follows: ； in, Represents the input matrix, dimension Compared to the high axis, the wide axis swaps the rows and columns of the input matrix. It is the length of the input matrix. It is the width of the input matrix. , , Represents a query, key, and value matrix. The dimension is , The dimension is , The dimension is , , , These represent the extracted corresponding features, with dimensions of respectively. , , ; Calculate the attention along the wide axis, with attention weight scores for all rows in each column: ； Softmax represents the normalization function. This represents the wide-axis attention feature of the output; By concatenating wide-axis attention and high-axis attention, attention is calculated separately in the width and height directions to extract global attention. Compared to non-local self-attention, this method of calculating attention separately for the width and height axes is more efficient and faster. Multimodal image features, after undergoing axis attention, acquire more spatially expressive features. , To obtain better channel dimension features, the features extracted by axis attention are transformed into separable channel attention modules after dimension-corrected convolution. The main operations of these modules include separable convolution, global pooling, fully connected operations, normalization, and dimension restoration.

5. The multimodal image fusion method based on a deep encoder-decoder axis interaction attention network as described in claim 3, characterized in that, The separable channel attention module: Image feature dimensions are compressed using separable convolution and global pooling operations, concentrating spatial feature information into channels and aggregating it into a single vector. This vector is then incorporated into a fully connected layer for feature full connectivity, while an activation function is applied to increase its non-linearity. Finally, softmax is used for normalization to ensure that the sum of the calculated weight components is 1. For the input features... The calculation process for the separation channel attention module is as follows: ； in, This indicates a separable operation, using a convolution with a kernel of 1. Indicates global average pooling. This indicates a fully connected operation. It's the activation function; here, ReLU is used. This represents the Softmax activation function. This represents the corresponding attention weight for capture, which ranges from [0,1]. The features extracted by the separable channel attention are then passed to the modal interaction attention module.

6. The multimodal image fusion method based on a deep encoder-decoder axis-interaction attention network as described in claim 3, characterized in that, The modal interaction attention module: Multi-head interactive attention is used to model long-distance dependencies between different modalities. For features extracted by the encoder, ( ), This indicates the number of layers in the encoder that extract features; it consists of four layers. Used to distinguish modes , This represents a visible and infrared multimodal image, with a size of [size missing]. , Indicates the feature map height. Indicates the width of the feature map. This represents the number of feature map channels, and the initial extraction is performed using a size of [size missing]. Local window, This indicates the length and width of a local window. Reintegrated into , This represents the dimension of the feature within each local window. The number of windows is determined by the total number of pixels in the feature map. Divide by the number of pixels in a single window After obtaining the result, a self-attention extraction operation is performed on each local window, and a weight learning matrix is ​​then used. For local window features Perform mapping. Represents the local window feature matrix. Represents the real number field. It is the dimension of the matrix, that is, we have OK, The real matrix of columns, to obtain the query ,key ,value , subscript This represents a visible light diagram, with subscripts indicating the direction of light. Infrared image representation: ； ； Then calculate and another mode The dot product is then normalized using the Softmax function to obtain the attention weight values. For each attention head, the calculation process is as follows: ； in, Indicator key Dimensions Is a key Dimension The square root is used to avoid dimensionality. An excessively large value causes the gradient of the softmax function to vanish. Indicates query Perform a dot product operation on the transpose of the bond to obtain a matrix representation. and Similarity score, This represents the output of a single attention head. The obtained self-attention is then extended to multi-head interactive self-attention to extract different attention distributions and capture information from different dimensions. Due to the above extraction , and The information comes from two modalities, and the cross-modal extraction process can be represented as follows: ； ； Among them, the visible light mode With infrared mode and Perform cross-modal information aggregation. This indicates multi-head interactive attention. The representation layer is normalized, and residual connections are introduced to retain the initial information of the visible light modes in order to establish the connection and interaction between different modes and keep the input information from being lost. ； ； in, and This indicates the intermediate features of the multi-head interactive attention mechanism extracted from visible light and infrared light. Indicates a feedforward network. Representation layer normalization, and This represents the extracted multi-head interaction attention features; at last, and Channel concatenation is performed, and dimensionality correction is applied using convolution to obtain fused features: ； in This indicates the fusion feature.

7. The multimodal image fusion method based on a deep encoder-decoder axis interaction attention network as described in claim 1, characterized in that, the Decoder: Composed of decoding block 1, decoding block 2 and decoding block 3. The decoder has the same structure as the encoder's convolutional layer and encoding layer, but omits the Transformer feature extraction part and introduces cross-layer skip connections and residual networks to make full use of the feature decoding of each layer. Finally, the fused image is obtained by gradually extracting features through the decoder.

8. The multimodal image fusion method based on a deep encoder-decoder axis-interactive attention network as described in claim 1, characterized in that, After the image fusion is completed, a loss calculation is performed using a combination of pixel loss, structural similarity loss, and gradient loss. The loss function applies pixel loss, structural similarity loss, and gradient loss, and the calculation process of the loss function is as follows: ； in, Indicates total training loss. Indicates pixel loss, Represents structural similarity loss. For gradient loss; Pixel loss is used to constrain the similarity in pixel values ​​between the fused image and the source image, and its calculation can be expressed as: ； in, The corresponding pixel values ​​of the input image. To output the pixel values ​​of the image, This corresponds to the position of the pixel. Indicates the number of iterations; Structural similarity loss is used to measure the structural similarity between the fused image and the source image, and its calculation can be expressed as: ； ； in, It is the input image. It outputs an image. It is a structural similarity function. , The mean of the input and output images. , A standard table for input and output images. For the input image and output image covariance, , , This is a constant term set to avoid a denominator of 0, where This is a constant term set to avoid a denominator of 0. , , ; Gradient loss is used to preserve edge and texture details of an image, and its calculation can be expressed as: ； in, It is an image gradient, It is an image The gradient.

Images