Multi-modal image fusion method based on multi-scale reversible-mutual attention mechanism

Abstract

The invention discloses a multi-modal image fusion method based on a multi-scale reversible-mutual attention mechanism. The method comprises the following steps: firstly, sending an infrared image and a visible light image into a ResInvNet encoder module, and extracting multi-scale features; then the extracted infrared and visible light multi-scale features are sent to a cross-modal feature fusion module to generate fusion features, and a fused image is output through a decoder module; and constructing a loss function optimization multi-modal image fusion model, and finally realizing image fusion based on the optimized multi-modal image fusion model. According to the ResInvNet, high-frequency beneficial information of the features can be reserved as much as possible in the process of extracting the deeper-level features by the model. The cross-modal feature fusion module is based on a BSCA module and an AttenFuse module, can efficiently obtain beneficial features complementary among modals, generates a required fusion image for a multi-modal downstream task, and further guarantees the stability of a visual algorithm.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and pattern recognition, and relates to the improvement of an infrared image and a visible light image fusion algorithm, and the realization and application of a reversible encoder structure and a mutual attention fusion structure; specifically, it relates to a multimodal image fusion method based on a multi-scale reversible-mutual attention mechanism. Background Art

[0002] With the widespread application of artificial intelligence, traditional processing methods for visible light images, often due to insufficient lighting conditions, glare and other problems, have been guaranteed to ensure stable prediction needs in extreme cases. The development of sensing technology has made the acquisition of multimodal data more convenient. Image data of different modalities have different perception and distribution characteristics. Therefore, multimodal image fusion has become a research hotspot. However, multimodal image processing faces many challenges. The inherent barriers between different modalities, such as differences in distribution and scale, seriously affect the model's use of multimodal features. Therefore, an effective method is needed to address these challenges.

[0003] In practical applications, the scale of objects in images is not constant. As the number of convolutional neural network layers increases, the model extracts more advanced semantic information, but due to information loss caused by downsampling, it is inevitable to lose detailed, edge-related high-level semantic features. Therefore, introducing features of different sizes for fusion and focusing on the efficiency of backward transmission of high-frequency features is one of the key issues to ensure efficient image fusion algorithms.

[0004] With the development of deep learning, the introduction of mutual attention module and reversible neural network provides new ideas for solving the above problems. In recent years, Transformer has been widely used in the field of computer graphics due to its excellent global modeling ability. Sending two inputs from different sources as query vector (Query) and key / value vector (Key / Value) into the mutual attention module can automatically learn the correlation between different inputs and dynamically adjust the weight distribution of features according to their importance, so that the model can selectively focus on the information that is most helpful for the current task. During forward propagation, the reversible neural network maps the input features to the hidden representation space through a series of reversible transformation operations, and then maps the hidden representation space back to the original input space through inverse transformation, reducing the information loss in the information transmission process, thereby effectively retaining the high-frequency features in the original image.

[0005] The excellent characteristics of the mutual attention module and the reversible neural network meet the challenges of cross-modal feature fusion tasks. Based on this, the present invention provides a multimodal image fusion method based on a multi-scale reversible-mutual attention mechanism. Summary of the invention

[0006] The purpose of the present invention is to solve the problem that traditional convolutional neural networks are difficult to efficiently utilize multimodal features due to inherent barriers between different modalities, among which high-frequency feature loss is the key to solving this problem.

[0007] In order to achieve the above object, the present invention proposes a multimodal image fusion method based on a multi-scale reversible-mutual attention mechanism. The method can combine the useful information of multimodal source images to generate a fused image with richer information.

[0008] A multimodal image fusion method based on a multi-scale reversible-mutual attention mechanism comprises the following steps:

[0009] Step 1: Send the infrared image and the visible light image to two parameter-sharing ResInvNet encoder modules based on convolutional neural networks and reversible neural networks to extract multi-scale features;

[0010] Step 2: The extracted infrared and visible light multi-scale features are sent to a cross-modal feature fusion module based on a convolutional neural network and Transformer to generate fused features;

[0011] Step 3: Send the fused features to the decoder module constructed by the convolutional neural network and output the fused image;

[0012] Step 4: Construct a loss function to optimize the multimodal image fusion model composed of the above-mentioned ResInvNet encoder, cross-modal fusion module and decoder module.

[0013] Step 5: Implement image fusion based on the optimized multimodal image fusion model.

[0014] The infrared image and visible light image to be fused are input into the optimized multimodal image fusion model, and after model inference, the fused image is output.

[0015] Furthermore, the ResInvNet encoder module includes five ResInv Blocks connected in series, and the outputs of each level are defined as the first layer features F in the order of connection. 1 To the 5th layer feature F 5 The ResInv Block takes the input feature F i Split into high-frequency features evenly by channel dimension and low frequency characteristics The two parts are sent to the high-frequency branch and the low-frequency branch respectively to process the high-frequency and low-frequency parts of the features. The superscript i refers to the level of the feature input, ranging from 0 to 4. When i = 0, the input of ResInv Block is an infrared image or a visible light image.

[0016] In order to retain more high-frequency information about edges and textures, the high-frequency branch adds the high-frequency features and the low-frequency features mapped by convolution pixel by pixel, and outputs the enhanced high-frequency features. Its mathematical expression is as follows:

[0017]

[0018] Among them, Convs represents a convolution block, which is composed of two 2D convolutions with 3×3 kernels, stride 1, and Padding 1, and a BatchNorm and Relu activation function connected in series.

[0019] The low-frequency branch optimizes the low-frequency features based on the enhanced high-frequency features output by the high-frequency branch to generate enhanced low-frequency features Its mathematical expression is as follows:

[0020]

[0021] in, It is the enhanced high-frequency feature guidance after convolution mapping, which is used to obtain the intermediate features after filtering the effective features in the low-frequency features. exp represents the exponential activation function.

[0022] Furthermore, the enhanced high-frequency features and low-frequency features are concatenated according to the channel dimension to obtain the output F of the ResInv Block. i+1 .

[0023] More specifically, when the input of the ResInvNet encoder module is an infrared image, the obtained multi-scale features are labeled as Correspondingly, when the input of the ResInvNet encoder module is a visible light image, the obtained multi-scale features are marked as Here, i=1,2,3,4,5.

[0024] Furthermore, the cross-modal feature fusion module includes two modules: Bidirectional Supervised CrossAttention (BSCA) and Attention-based Fuse (AttenFuse).

[0025] The specific processing process of the cross-modal feature fusion module is as follows:

[0026] 1) The processing of multi-scale features is to process the features of the 3rd to 5th layers. More specifically, the processing strategies for the 3rd and 4th layer features and the 5th layer features are different according to the different information that the features of different levels focus on;

[0027] 2) The processing strategy of the third and fourth layer features is to convert the infrared and visible light features of the corresponding layers Send it to the BSCA module to obtain enhanced infrared and visible light features Next, the enhanced multimodal features are used as the input of the Attention-based Fuse module for feature fusion to generate fused features Where i = 3, 4;

[0028] 3) The processing strategy of the fifth layer features is to combine the infrared and visible light features of the fifth layer Directly send it to the Attention-based Fuse module for feature fusion to generate the 4th layer of fusion features

[0029] Furthermore, the BSCA module is built on the basis of Transformer and includes two independent CrossAttention modules, which are used to enhance the input visible light modal features and infrared modal features respectively. Entered as a query, infrared modal features Input as Key and Value, output the visible light modal features enhanced by the infrared modal features On the contrary, for the Cross Attention module that enhances infrared features, the infrared features As a query, visible light mode As Key and Value, generate

[0030] Furthermore, the AttenFuse module is constructed by a convolutional neural network, and the specific processing process is as follows: 1) the features of the two modalities at the same level of the input are spliced ​​along the channel dimension; 2) they are sent to the convolution block, where the number of output channels is 1 / 2 of the number of input channels; 3) fusion features containing cross-modal information are obtained

[0031] Furthermore, the multi-scale fusion features generated by the cross-modal feature fusion module are In the reverse direction, upsampling is performed layer by layer, and the features of the previous layer are concatenated and convolved according to the channel dimension to generate features that fuse multi-scale information. Its specific mathematical expression is as follows:

[0032]

[0033] in, It is the final generated fusion feature containing the multi-scale information of the 3rd to 5th layers, which serves as the input of the decoder module; UP represents the upsampling operation, and Concat represents the concatenation operation according to the channel dimension.

[0034] Furthermore, the decoder module is constructed by a convolutional neural network, and its output should be H×W×3, where H is the height of the input image, W is the width of the input image, and 3 represents the number of channels required for RGB color images.

[0035] Furthermore, the loss function used to optimize the model is the cross entropy loss function, which includes two parts: supervision of the final output and supervision of the enhanced features generated by the BSCA module. Specifically, the supervision goal of the final output is to make the output as similar as possible to the ground truth provided by the dataset, and the supervision goal of the BSCA module is to make and and As close as possible.

[0036] The beneficial effects of the present invention are as follows:

[0037] The present invention performs cross-modal image fusion on images of infrared modalities and visible light modalities acquired from different sensors to generate a fused image containing the beneficial features of each modality. In this method, ResInvNet, which is constructed based on convolutional neural networks and reversible neural networks, can retain as much high-frequency beneficial information of features as possible in the process of extracting deeper-level features from the model; the Bidirectional Supervised CrossAttention module constructed based on Transformer can reduce the differences between modalities and achieve cross-modal alignment; the Attention-based Fuse module constructed based on convolutional neural networks can achieve cross-modal feature fusion. Combined with the modules, this method can efficiently obtain complementary beneficial features between modalities, generate the required fused images for multimodal downstream tasks, and thus ensure the stability of the visual algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a model network structure diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, the description of known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present invention. The present invention is described in detail below according to the accompanying drawings, and the purpose and effect of the present invention will become more obvious.

[0040] Figure 1 This is a model network structure diagram of an embodiment of the present invention. The network mainly consists of three structures: feature extraction encoder, cross-modal feature fusion and decoder, and mainly includes three modules: ResInvNet, Bidirectional Supervised CrossAttention and Attention-based Fuse. Among them, T represents infrared modality, V represents visible light modality, Pred represents the output image of the method, Represents the output features of the i-th layer ResInv Block of M modality, i = 1, 2, 3, 4, 5. Convs represents a convolution block, which is composed of two 2D convolutions with 3×3 kernels, stride 1, and padding 1, BatchNorm and Relu activation functions connected in series, exp represents an exponential activation function, and UP represents an upsampling operation.

[0041] Combination Figure 1 The multi-modal image fusion method based on multi-scale reversible-mutual attention mechanism comprises the following steps:

[0042] Step 1: Send the infrared image and the visible light image to two parameter-sharing ResInvNet encoder modules based on convolutional neural networks and reversible neural networks to extract multi-scale features;

[0043] The ResInvNet encoder module contains 5 ResInv Blocks connected in series, and the output of each level is defined as the first layer feature F according to the connection order. 1 To the 5th layer feature F 5 The ResInv Block takes the input feature F i Split into high-frequency features evenly by channel dimension and low frequency characteristics The two parts are sent to the high-frequency branch and the low-frequency branch respectively to process the high-frequency and low-frequency parts of the features. The superscript i refers to the level of the feature, ranging from 0 to 4. When i = 0, the input of ResInvBlock is an infrared image or a visible light image.

[0044] In order to retain more high-frequency information about edges and textures, the high-frequency branch adds the high-frequency features and the low-frequency features mapped by convolution pixel by pixel, and outputs the enhanced high-frequency features. Its mathematical expression is as follows:

[0045]

[0046] Among them, Convs represents a convolution block, which is composed of two 2D convolutions with 3×3 kernels, stride 1, and Padding 1, and a BatchNorm and Relu activation function connected in series.

[0047] The low-frequency branch optimizes the low-frequency features based on the enhanced high-frequency features output by the high-frequency branch to generate enhanced low-frequency features Its mathematical expression is as follows:

[0048]

[0049] in, It is the intermediate feature obtained after filtering out the effective features in the low-frequency features under the guidance of the enhanced high-frequency features of the convolutional mapping. exp represents the exponential activation function.

[0050] Furthermore, the enhanced high-frequency features and low-frequency features are concatenated according to the channel dimension to obtain the output F of the ResInv Block. i+1 .

[0051] More specifically, when the input of the ResInvNet encoder module is an infrared image, the obtained multi-scale features are labeled as Correspondingly, when the input of the ResInvNet encoder module is a visible light image, the obtained multi-scale features are marked as Here, i=1,2,3,4,5.

[0052] Step 2: The extracted infrared and visible light multi-scale features are sent to a cross-modal feature fusion module based on a convolutional neural network and Transformer to generate fused features;

[0053] The cross-modal feature fusion module includes two modules: Bidirectional Supervised Cross Attention (BSCA) and Attention-based Fuse (AttenFuse).

[0054] The specific processing process of the cross-modal feature fusion module is as follows:

[0055] 1) The processing of multi-scale features is to process the features of the 3rd to 5th layers. More specifically, the processing strategies for the 3rd and 4th layer features and the 5th layer features are different according to the different information that the features of different levels focus on;

[0056] 2) The processing strategy of the third and fourth layer features is to convert the infrared and visible light features of the corresponding layers Send it to the BSCA module to obtain enhanced infrared and visible light features Next, the enhanced multimodal features are used as the input of the Attention-based Fuse module for feature fusion to generate fused features Here, i=3,4;

[0057] 3) The processing strategy of the fifth layer features is to combine the infrared and visible light features of the fifth layer Directly send it to the Attention-based Fuse module for feature fusion to generate the 4th layer of fusion features

[0058] The BSCA module is built on the basis of Transformer and contains two independent Cross Attention modules, which are used to enhance the input visible light modal features and infrared modal features respectively. Entered as a query, infrared modal features Input as Key and Value, output the visible light modal features enhanced by the infrared modal features On the contrary, the CrossAttention module that enhances infrared features As a query, visible light mode As Key and Value, generate

[0059] The AttenFuse module is constructed by a convolutional neural network. The specific processing process is as follows: 1) The features of the two modalities at the same level of the input are spliced ​​along the channel dimension; 2) They are sent to the convolution block, where the number of output channels is 1 / 2 of the number of input channels; 3) The fusion features containing cross-modal information are obtained

[0060] Step 3: Send the fused features to the decoder module constructed by the convolutional neural network and output the fused image;

[0061] The multi-scale fusion features generated by the cross-modal feature fusion module are In the reverse direction, upsampling is performed layer by layer, and the features of the previous layer are concatenated and convolved according to the channel dimension to generate features that fuse multi-scale information. Its specific mathematical expression is as follows:

[0062]

[0063] in, It is the final generated fusion feature containing the multi-scale information of the 3rd to 5th layers, which serves as the input of the decoder module; UP represents the upsampling operation, and Concat represents the concatenation operation according to the channel dimension.

[0064] The decoder module is constructed by a convolutional neural network. More specifically, in the embodiment, the decoder module includes three layers of serially connected decoding blocks, and the decoder blocks are connected by upsampling. Each decoder block includes two consecutive convolutional layers with ReLU activation function, and its output should be H×W×3, where H is the height of the input image, W is the width of the input image, and 3 represents the number of 3 channels required for RGB color images.

[0065] Step 4: Construct a loss function to optimize the multimodal image fusion model composed of the above-mentioned ResInvNet encoder, cross-modal fusion module and decoder module.

[0066] The loss function used to optimize the model is the cross entropy loss function, which includes two parts: supervision of the final output and supervision of the enhanced features generated by the BSCA module. Specifically, the supervision goal of the final output is to make the output as similar as possible to the ground truth provided by the dataset, and the supervision goal of the BSCA module is to make and and As close as possible.

[0067] Step 5: Image fusion is realized based on the optimized multimodal image fusion model. The input of the model is the infrared image and the visible light image to be fused. After model reasoning, the fused image is output.

[0068] It should be understood that the above specific embodiments of the present invention are only used to illustrate or explain the principles of the present invention, and do not constitute a limitation of the present invention. Therefore, any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present invention should be included in the protection scope of the present invention. In addition, the appended claims of the present invention are intended to cover all changes and modifications that fall within the scope and boundaries of the appended claims, or the equivalent forms of such scope and boundaries.

Claims

1. A multimodal image fusion method based on a multi-scale reversible-mutual attention mechanism, characterized in that: The following steps are involved: Step 1: Send the infrared image and the visible light image to two parameter-sharing ResInvNet encoder modules based on convolutional neural networks and reversible neural networks to extract multi-scale features; Step 2: The extracted infrared and visible light multi-scale features are sent to a cross-modal feature fusion module based on a convolutional neural network and Transformer to generate fused features; Step 3: Send the fused features to the decoder module constructed by the convolutional neural network and output the fused image; Step 4, construct a loss function for optimizing the multimodal image fusion model composed of the above-mentioned ResInvNet encoder, cross-modal fusion module and decoder module; Step 5, realizing image fusion based on the optimized multimodal image fusion model; The infrared image and visible light image to be fused are input into the optimized multimodal image fusion model, and after model inference, the fused image is output.

2. According to claim 1, a multimodal image fusion method based on a multi-scale reversible-mutual attention mechanism is characterized in that: The ResInvNet encoder module contains 5 ResInv Blocks connected in series, and the output of each level is defined as the first layer feature F according to the connection order. 1 To the 5th layer feature F 5 ; The ResInv Block inputs the feature F i Split into high-frequency features evenly by channel dimension and low frequency characteristics The two parts are sent to the high-frequency branch and the low-frequency branch respectively to process the high-frequency and low-frequency parts of the features; the superscript i refers to the level of the feature input, ranging from 0 to 4 five integers; when i = 0, the input of ResInv Block is an infrared image or a visible light image; The high-frequency branch adds the high-frequency features to the low-frequency features mapped by convolution pixel by pixel, and outputs the enhanced high-frequency features. Its mathematical expression is as follows: Among them, Convs represents a convolution block, which is composed of two 2D convolutions with 3×3 kernels, stride 1, and padding 1, and a BatchNorm and Relu activation function connected in series; The low-frequency branch optimizes the low-frequency features based on the enhanced high-frequency features output by the high-frequency branch to generate enhanced low-frequency features Its mathematical expression is as follows: in, It is the enhanced high-frequency feature guidance after convolution mapping, which is used to obtain the intermediate features after filtering the effective features in the low-frequency features. exp represents the exponential activation function; Furthermore, the enhanced high-frequency features and low-frequency features are concatenated according to the channel dimension to obtain the output F of the ResInv Block. i+1 .

3. The multimodal image fusion method based on multi-scale reversible-mutual attention mechanism according to claim 1 or 2, characterized in that: The cross-modal feature fusion module includes two modules: BSCA and AttenFuse; The specific processing process of the cross-modal feature fusion module is as follows: 1) The processing of the multi-scale features is to process the features of the 3rd to 5th layers; more specifically, according to the different information focused on by the features of different levels, the processing strategies for the features of the 3rd and 4th layers and the features of the 5th layer are different; 2) The processing strategy of the third and fourth layer features is to convert the infrared and visible light features of the corresponding layers Send it to the BSCA module to obtain enhanced infrared and visible light features Next, the enhanced multimodal features are used as the input of the Attention-based Fuse module for feature fusion to generate fused features Where i = 3, 4; 3) The processing strategy of the fifth layer features is to combine the infrared and visible light features of the fifth layer Directly send it to the Attention-based Fuse module for feature fusion to generate the 4th layer of fusion features 4. The multimodal image fusion method based on multi-scale reversible-mutual attention mechanism according to claim 3 is characterized in that: The BSCA module is built on the basis of Transformer and includes two independent Cross Attention modules, which are used to enhance the input visible light modal features and infrared modal features respectively. For the CrossAttention module that enhances visible light features, the visible light features Entered as a query, infrared modal features Input as Key and Value, output the visible light modal features enhanced by the infrared modal features On the contrary, for the CrossAttention module that enhances infrared features, the infrared features As a query, visible light mode As Key and Value, generate 5. The multimodal image fusion method based on multi-scale reversible-mutual attention mechanism according to claim 3, characterized in that: The AttenFuse module is constructed by a convolutional neural network. The specific processing process is as follows: 1) The features of the two modalities at the same level of the input are spliced ​​along the channel dimension; 2) They are sent to the convolution block, where the number of output channels is 1 / 2 of the number of input channels; 3) The fusion features containing cross-modal information are obtained 6. The multimodal image fusion method based on multi-scale reversible-mutual attention mechanism according to claim 5, characterized in that: The multi-scale fusion features generated by the cross-modal feature fusion module are In the reverse direction, upsampling is performed layer by layer, and the features of the previous layer are concatenated and convolved according to the channel dimension to generate features that fuse multi-scale information. Its specific mathematical expression is as follows: in, It is the final generated fusion feature containing the multi-scale information of the 3rd to 5th layers, which serves as the input of the decoder module; UP represents the upsampling operation, and Concat represents the concatenation operation according to the channel dimension.

7. The multimodal image fusion method based on multi-scale reversible-mutual attention mechanism according to claim 6, characterized in that: The decoder module is constructed by a convolutional neural network, and its output should be H×W×3, where H is the height of the input image, W is the width of the input image, and 3 represents the number of channels required for RGB color images.

8. The multimodal image fusion method based on multi-scale reversible-mutual attention mechanism according to claim 7, characterized in that: The loss function used to optimize the model is the cross entropy loss function, which includes two parts: supervision of the final output and supervision of the enhanced features generated by the BSCA module. Specifically, the supervision goal of the final output is to make the output as similar as possible to the true value label provided by the dataset, and the supervision goal of the BSCA module is to make and and As close as possible.

Images