A multi-modal image fusion method based on a multi-scale reversible cross-attention mechanism

Abstract

The application discloses a kind of multi-modal image fusion methods based on multi-scale reversible-inter attention mechanism.First, infrared image and visible light image are sent into ResInvNet encoder module, and multi-scale features are extracted;Then the extracted infrared and visible light multi-scale features are sent into cross-modal feature fusion module, to generate fusion features, and output the fused image through the decoder module;Loss function is constructed to optimize multi-modal image fusion model, and finally based on the optimized multi-modal image fusion model, image fusion is realized.ResInvNet can retain as much high-frequency beneficial information of features as possible during the process of extracting deeper level features by the model.The cross-modal feature fusion module based on BSCA module and AttenFuse module can efficiently obtain beneficial features complementary between modalities, generate the required fusion image for multi-modal downstream tasks, and thus ensure the stability of visual algorithms.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of artificial intelligence and pattern recognition, and relates to improvement of an infrared image and visible light image fusion algorithm, and implementation and application of a reversible encoder structure and a mutual attention fusion structure; in particular, relates to a multi-modal image fusion method based on a multi-scale reversible-mutual attention mechanism. BACKGROUND

[0002] With the wide application of artificial intelligence, the traditional processing method for visible light images often fails to ensure stable prediction under extreme conditions due to insufficient light, glare and other problems. The development of sensing technology has gradually made it convenient to obtain multi-modal data. Different modal image data has different perception and distribution characteristics, so multi-modal image fusion has become a research hotspot. However, multi-modal image processing faces many challenges. The inherent barriers between different modalities, such as distribution, scale and other differences, seriously affect the utilization of multi-modal features by the model. Therefore, an effective method is needed to solve these challenges.

[0003] In practical applications, the size of the object in the image is not fixed. With the deepening of the convolutional neural network layer, the model extracts higher-level semantic information, but inevitably loses the detailed and edge-related high-level semantic features due to the information loss caused by down-sampling. Therefore, introducing different size features for fusion and focusing on the efficiency of the backward transmission of high-frequency features is one of the key problems to ensure an efficient image fusion algorithm.

[0004] With the development of deep learning, the mutual attention module and the reversible neural network provide a new way to solve the above problems. In recent years, the Transformer has been widely applied to the field of computer images due to its excellent global modeling ability. By sending two inputs of different sources into the mutual attention module as the query vector (Query) and the key / value vector (Key / Value), the mutual attention module can automatically learn the correlation between different inputs and dynamically adjust the weight distribution of the features according to their importance, so that the model can selectively focus on the information that is most helpful to the current task. During forward propagation, the reversible neural network maps the input features to a hidden representation space through a series of reversible transformation operations, and then maps the hidden representation space back to the original input space through inverse transformation, reducing the information loss in the information transmission process, thereby effectively preserving the high-frequency features in the original image.

[0005] The excellent characteristics of the mutual attention module and the reversible neural network meet the challenges required by the cross-modal feature fusion task. Based on this, the present application provides a multi-modal image fusion method based on a multi-scale reversible-mutual attention mechanism. SUMMARY

[0006] The purpose of the present application is to solve the problem that traditional convolutional neural networks cannot efficiently utilize multi-modal features due to inherent barriers between different modalities, where high-frequency feature loss is the key to solving this problem.

[0007] To achieve the purpose, the present application proposes a multi-modal image fusion method based on a multi-scale reversible-attention mechanism.

[0008] A multi-modal image fusion method based on a multi-scale reversible-attention mechanism includes the following steps:

[0009] Step 1, the infrared image and the visible light image are sent into a ResInvNet encoder module constructed based on a convolutional neural network and a reversible neural network with shared parameters to extract multi-scale features;

[0010] Step 2, the extracted infrared and visible light multi-scale features are sent into a cross-modal feature fusion module constructed based on a convolutional neural network and a Transformer to generate fusion features;

[0011] Step 3, the fusion features are sent into a decoder module constructed by a convolutional neural network to output the fused image;

[0012] Step 4, a loss function is constructed to optimize the multi-modal image fusion model composed of the above ResInvNet encoder, cross-modal fusion module and decoder module.

[0013] Step 5, image fusion is realized based on the optimized multi-modal image fusion model.

[0014] The infrared image and the visible light image to be fused are input into the optimized multi-modal image fusion model, and the fused image is output after model inference.

[0015] Further, the ResInvNet encoder module includes 5 ResInv Blocks connected in series, and the outputs of each level are defined as the 1st layer feature F 1 to the 5th layer feature F 5 in connection order. The ResInv Block splits the input feature F i into high-frequency feature and low-frequency feature two parts in the channel dimension, and sends them into the high-frequency branch and the low-frequency branch to process the high-frequency and low-frequency parts of the feature. Wherein, the superscript i indicates the level of feature input, and the range is 0-4 five integers. When i=0, the input of the ResInv Block is the infrared image or the visible light image.

[0016] To preserve more high-frequency information about the edge and texture, the high-frequency branch adds the high-frequency features and the low-frequency features mapped by convolution pixel by pixel, and outputs the enhanced high-frequency features The mathematical expression is as follows:

[0017]

[0018] Wherein, Convs represents a convolution block, which is composed of two 2D convolutions with a kernel of 3x3, a stride of 1 and padding of 1, BatchNorm and Relu activation function in series.

[0019] The low-frequency branch optimizes the low-frequency features based on the enhanced high-frequency features output by the high-frequency branch, and generates enhanced low-frequency features The mathematical expression is as follows:

[0020]

[0021] Wherein, The enhanced high-frequency features mapped by convolution guide the intermediate features obtained after screening the effective features in the low-frequency features, and exp represents the exponential activation function.

[0022] Further, the enhanced high-frequency features and the low-frequency features are spliced in the channel dimension to obtain the output F of the ResInv Block i+1 .

[0023] More specifically, when the input of the ResInvNet encoder module is an infrared image, the obtained multi-scale features are marked as Correspondingly, when the input of the ResInvNet encoder module is a visible light image, the obtained multi-scale features are marked as Here, i = 1, 2, 3, 4, 5.

[0024] Further, the cross-modal feature fusion module includes two modules of Bidirectional Supervised Cross Attention (BSCA) and Attention-based Fuse (AttenFuse).

[0025] The specific processing process of the cross-modal feature fusion module is as follows:

[0026] 1) The processing of the multi-scale features is to process the features of the 3rd to 5th layers. More specifically, according to the different information focused by different levels of features, the processing strategies of the 3rd and 4th layer features and the 5th layer features are not the same;

[0027] 2) the processing strategy of the 3rd and 4th layer features, specifically the infrared and visible light features of the corresponding layer level into the BSCA module to obtain enhanced infrared and visible light features Then, the enhanced multi-modal features are taken as the input of the Attention-based Fuse module for feature fusion to generate fused features where i = 3, 4;

[0028] 3) the processing strategy of the 5th layer features, specifically the infrared and visible light features of the 5th layer are directly sent into the Attention-based Fuse module for feature fusion to generate the 4th layer fused features

[0029] Further, the BSCA module is constructed based on the Transformer and contains two independent Cross Attention modules, which are respectively used to enhance the input visible light modal features and infrared modal features. For the Cross Attention module for enhancing the visible light features, the visible light features are taken as the Query, the infrared modal features are taken as the Key and Value, and the output is the visible light modal features enhanced by the guidance of the infrared modal features Conversely, for the Cross Attention module for enhancing the infrared features, the infrared features are taken as the Query, the visible light modal are taken as the Key and Value, and the output is the infrared features enhanced by the guidance of the visible light modal

[0030] Further, the AttenFuse module is constructed by a convolutional neural network, and the specific processing process is as follows: 1) the features of the two modalities of the same layer level are spliced along the channel dimension; 2) sent into the convolution block, where the output channel number is 1 / 2 of the input channel number; 3) the fused features containing cross-modal information are obtained

[0031] Further, the multi-scale fused features generated by the above cross-modal feature fusion module are reversely performed layer by layer upsampling, channel dimension splicing with the previous layer fused features, and convolution mapping to generate features fused with multi-scale information The specific mathematical expression is as follows:

[0032]

[0033] wherein, is the final generated fusion feature containing the 3-5 layer multi-scale information, which is the input of the decoder module; UP represents the up-sampling operation, and Concat represents the channel dimension splicing operation.

[0034] Further, the decoder module is constructed by a convolutional neural network, and the output thereof should be HxWx3, wherein H is the height of the input image, W is the width of the input image, and 3 represents the required 3 channel numbers of the RGB color image.

[0035] Further, the loss function for optimizing the model is a cross-entropy loss function, which includes two parts of supervision of the final output result and the supervision of the enhanced feature generated by the BSCA module. Specifically, the supervision target of the final output is to make the output as similar as possible to the ground truth label provided by the data set, and the supervision target of the BSCA module is to make the enhanced feature as close as possible to the feature of the input image. and and as close as possible.

[0036] The beneficial effects of the present application are as follows:

[0037] The present application faces the images of infrared modal and visible light modal acquired from different sensors, performs cross-modal image fusion, and generates a fusion image containing the advantageous features of each modal. In the method, the ResInvNet constructed based on the convolutional neural network and the reversible neural network can retain as much high-frequency beneficial information of the features as possible in the process of extracting deeper level features by the model; the Bidirectional Supervised CrossAttention module constructed based on the Transformer can reduce the difference between the modes and realize the alignment of the cross-modal; and the Attention-based Fuse module constructed based on the convolutional neural network can realize the feature fusion of the cross-modal. In combination with the modules, the present method can efficiently acquire the beneficial features complementary between the modes, generate the required fusion image for the multi-modal downstream task, and further ensure the stability of the visual algorithm. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 is the model network structure diagram of the embodiment of the present application. DETAILED DESCRIPTION

[0039] For the purposes of the present application, the technical solutions and advantages will be more clearly and specifically described below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present application. In addition, in the following description, descriptions of well-known structures and techniques are omitted to avoid unnecessarily obscuring the concept of the present application. The present application will be described in detail below according to the accompanying drawings, and the objects and effects of the present application will become more apparent.

[0040] Figure 1 is a model network structure diagram of an embodiment of the present application, and the network mainly consists of three structures of a feature extraction encoder, a cross-modal feature fusion and a decoder, mainly including three modules of ResInvNet, Bidirectional Supervised CrossAttention and Attention-based Fuse. Among them, T represents the infrared modality, V represents the visible light modality, Pred represents the output image of the method, represents the output feature of the i-th layer ResInv Block of the M modality, i = 1, 2, 3, 4, 5. Convs represents a convolution block, which is composed of two-dimensional convolution with two convolution kernels of 3x3, step length of 1, padding of 1, BatchNorm and Relu activation function in series, exp represents an exponential activation function, and UP represents an up-sampling operation.

[0041] In combination Figure 1 , the multi-modal image fusion method based on the multi-scale reversible-attention mechanism includes the following steps:

[0042] Step 1, the infrared image and the visible light image are sent into a ResInvNet encoder module constructed based on a convolutional neural network and a reversible neural network with shared parameters, to extract multi-scale features;

[0043] The ResInvNet encoder module includes five ResInv Blocks connected in series, and the outputs of each level are defined as the first layer feature F 1 to the fifth layer feature F 5 in the order of connection. The ResInv Block splits the input feature F i into high-frequency feature and low-frequency feature two parts in the channel dimension, and sends them into the high-frequency branch and the low-frequency branch respectively to process the high-frequency and low-frequency parts of the feature. Among them, the superscript i indicates the level of the feature, and the range is five integers 0-4. When i = 0, the input of the ResInv Block is the infrared image or the visible light image.

[0044] To preserve more high-frequency information about the edge and texture, the high-frequency branch adds the high-frequency features and the low-frequency features mapped by convolution pixel by pixel, and outputs enhanced high-frequency features The mathematical expression is as follows:

[0045]

[0046] Wherein, Convs represents a convolution block, which is composed of two 2D convolutions with a kernel size of 3x3, a stride of 1, and padding of 1, a BatchNorm, and a Relu activation function in series.

[0047] The low-frequency branch optimizes the low-frequency features based on the enhanced high-frequency features output by the high-frequency branch, and generates enhanced low-frequency features The mathematical expression is as follows:

[0048]

[0049] Wherein, is the intermediate feature obtained by screening the effective features in the low-frequency features under the guidance of the enhanced high-frequency features mapped by convolution, and exp represents an exponential activation function.

[0050] Further, the enhanced high-frequency features and the low-frequency features are spliced in the channel dimension to obtain the output F of the ResInv Block i+1 .

[0051] More specifically, when the input of the ResInvNet encoder module is an infrared image, the obtained multi-scale features are marked as Correspondingly, when the input of the ResInvNet encoder module is a visible light image, the obtained multi-scale features are marked as Here, i = 1, 2, 3, 4, 5.

[0052] Step 2, send the extracted infrared and visible light multi-scale features into the cross-modal feature fusion module based on convolutional neural network and Transformer to generate fusion features;

[0053] The cross-modal feature fusion module includes two modules, Bidirectional Supervised Cross Attention (BSCA) and Attention-based Fuse (AttenFuse).

[0054] The specific processing process of the cross-modal feature fusion module is as follows:

[0055] 1) the processing of multi-scale features is to process the features of the 3rd to 5th layers. More specifically, the processing strategies of the 3rd, 4th and 5th layer features are different according to the different information focused by different levels of features;

[0056] 2) the processing strategy of the 3rd and 4th layer features is to send the infrared and visible light features corresponding to the level into the BSCA module to obtain enhanced infrared and visible light features Then, the enhanced multi-modal features are taken as the input of the Attention-based Fuse module for feature fusion to generate fused features Here, i = 3, 4;

[0057] 3) the processing strategy of the 5th layer feature is to directly send the infrared and visible light features of the 5th layer into the Attention-based Fuse module for feature fusion to generate the 4th layer fused features

[0058] The BSCA module is constructed based on Transformer and contains two independent Cross Attention modules, which are respectively used to enhance the visible light modal features and the infrared modal features of the input. Taking the Cross Attention module for enhancing the visible light features as an example, the visible light features are taken as the Query, the infrared modal features are taken as the Key and Value, and the visible light modal features enhanced by the guidance of the infrared modal features are output Conversely, the Cross Attention module for enhancing the infrared features takes the infrared features as the Query, the visible light modal as the Key and Value, and generates

[0059] The AttenFuse module is constructed by a convolutional neural network, and the specific processing process is as follows: 1) the features of the two modalities of the same level are spliced along the channel dimension; 2) input into a convolutional block, where the output channel number is 1 / 2 of the input channel number; 3) obtain the fused features containing cross-modal information

[0060] Step 3: send the fused features into the decoder module constructed by the convolutional neural network to output the fused image;

[0061] The multi-scale fused features generated by the above cross-modal feature fusion module ​​The up-sampling, the channel dimension splicing and the convolution mapping are performed layer by layer in reverse, to generate the fused multi-scale information feature The specific mathematical expression is as follows:

[0062]

[0063] wherein, is the final generated fused feature containing the multi-scale information of the 3rd-5th layers, as the input of the decoder module; UP represents the up-sampling operation, and Concat represents the channel dimension splicing operation.

[0064] The decoder module is constructed by a convolutional neural network. More specifically, in the embodiment, the decoder module contains three decoding blocks connected in series, and the decoding blocks are connected by up-sampling. Each decoding block contains two consecutive convolution layers and a ReLU activation function, and the output thereof should be HxWx3, wherein H is the height of the input image, W is the width of the input image, and 3 represents the required 3 channel numbers of the RGB color image.

[0065] Step 4, a loss function is constructed for optimizing the multi-modal image fusion model composed of the ResInvNet encoder, the cross-modal fusion module and the decoder module.

[0066] The loss function for optimizing the model is a cross-entropy loss function, which contains two parts of supervision of the final output result and the enhanced feature generated by the BSCA module. Specifically, the supervision target of the final output is to make the output as similar as possible to the ground truth label provided by the data set, and the supervision target of the BSCA module is to make the enhanced feature as close as possible to the ground truth label provided by the data set. and and as close as possible.

[0067] Step 5, based on the optimized multi-modal image fusion model, the image fusion is realized, the input of the model is the infrared image and the visible light image to be fused, and the fused image is output after the model inference.

[0068] It should be understood that the above specific embodiments of the present application are only used for illustrative or explanatory purposes of the principles of the present application, and do not constitute a limitation on the present application. Therefore, any modification, equivalent replacement, improvement, etc. made without departing from the spirit and scope of the present application shall be included in the protection scope of the present application. In addition, the appended claims of the present application are intended to cover all variations and modifications falling within the scope and boundary of the appended claims, or the equivalent forms of such scope and boundary.

Claims

1. A multi-modal image fusion method based on a multi-scale reversible cross-attention mechanism, characterized in that, The method comprises the following steps: Step 1: input the infrared image and the visible light image into a ResInvNet encoder module based on convolutional neural network and reversible neural network sharing two parameters to extract multi-scale features; Step 2: input the extracted infrared and visible light multi-scale features into a cross-modal feature fusion module based on convolutional neural network and Transformer to generate fusion features; Step 3: input the fusion features into a decoder module constructed by convolutional neural network to output the fused image; Step 4: construct a loss function for optimizing the multi-modal image fusion model composed of the ResInvNet encoder, the cross-modal fusion module and the decoder module; Step 5: realize image fusion based on the optimized multi-modal image fusion model; input the infrared image and the visible light image to be fused into the optimized multi-modal image fusion model, perform model inference, and output the fused image; The ResInvNet encoder module contains 5 ResInv Blocks connected in series, and the output of each level is defined as the 0th layer feature F in the order of connection. 0 To the 4th layer feature F 4 ; The ResInv Block input feature F i Split into high-frequency features evenly by channel dimension and low-frequency characteristics The two parts are sent to the high-frequency branch and the low-frequency branch respectively to process the high-frequency and low-frequency parts of the features; the superscript i refers to the level of the feature input, ranging from 0 to 4 five integers; when i = 0, the input of ResInvBlock is an infrared image or a visible light image; The high-frequency branch adds high-frequency features and low-frequency features mapped by convolution pixel by pixel, and outputs enhanced high-frequency features The mathematical expression is as follows: wherein Convs represents a convolutional block composed of two two-dimensional convolutions with a kernel of 3x3, a stride of 1 and padding of 1, BatchNorm and Relu activation function in series; The low-frequency branch optimizes the low-frequency feature based on the enhanced high-frequency feature output by the high-frequency branch, to generate an enhanced low-frequency feature The mathematical expression is as follows: wherein, is a convolution-mapped enhanced high-frequency feature guide for screening the intermediate features obtained after screening the effective features in the low-frequency features, and exp represents an exponential activation function. Further, the enhanced high-frequency features and the low-frequency features are spliced according to the channel dimension to obtain the output F of the ResInv Block i+1 .

2. The multi-scale reversible cross-attention mechanism based multi-modal image fusion method according to claim 1, characterized in that, the cross-modal feature fusion module comprises two modules of BSCA and AttenFuse; the specific processing process of the cross-modal feature fusion module is as follows: 1) the processing of the multi-scale features is to process the features of the 3rd to 5th layers; more specifically, the processing strategies for the features of the 3rd and 4th layers and the features of the 5th layer are different due to the different information focused by the features of different layers; 2) processing strategy of the 3rd and 4th layer features, specifically, the infrared and visible light features corresponding to the layer level into the BSCA module to obtain enhanced infrared and visible light features Then, the enhanced multi-modal features are taken as the input of the Attention-based Fuse module for feature fusion to generate fused features wherein i = 3, 4; 3) the processing strategy of the 5th layer features, specifically the infrared and visible light features of the 5th layer are fused directly into the Attention-based Fuse module for feature fusion to generate the 4th layer fusion features The BSCA module is constructed based on a Transformer, and contains two independent Cross Attention modules, which are respectively used for enhancing visible light modal features and infrared modal features of the input; for the Cross Attention module for enhancing the visible light features, the visible light features The infrared modal features are sent in as Query The visible light modal features enhanced by the infrared modal features are outputted Conversely, for the Cross Attention module for enhancing the infrared features, the infrared features The visible light modal features are sent in as Query The visible light modal features enhanced by the infrared modal features are outputted The AttenFuse module is constructed by a convolutional neural network, and the specific processing process is as follows: 1) the features of two modalities in the same layer of the input are spliced along the channel dimension; 2) input into a convolutional block, wherein the output channel number is 1 / 2 of the input channel number; 3) obtain the fusion features containing cross-modal information 3. The multi-scale reversible cross-attention mechanism based multi-modal image fusion method of claim 2, wherein, The multi-scale fusion features generated by the cross-modal feature fusion module i=3, 4, 5, perform upsampling, channel dimension splicing with the fusion features of the previous layer, and convolution mapping in reverse layer by layer to generate features fused with multi-scale information The specific mathematical expression is as follows: wherein, is the final generated fusion feature containing the multi-scale information of the 3-5 layers, as the input of the decoder module; UP represents the up-sampling operation, and Concat represents the channel dimension splicing operation.

4. The multi-scale reversible cross-attention mechanism based multi-modal image fusion method of claim 3, characterized in that, the decoder module is constructed by convolutional neural network, and the output thereof should be HxWx3, wherein H is the height of the input image, W is the width of the input image, and 3 represents the required three channel numbers of the RGB color image.

5. The multi-scale reversible cross-attention mechanism based multi-modal image fusion method according to claim 4, characterized in that, The loss function for optimizing the model is a cross-entropy loss function, which includes two parts of supervision on the final output result and the enhanced features generated by the BSCA module; specifically, the supervision target for the final output is to make the output as similar as possible to the true value label provided by the data set, and the supervision target for the BSCA module is to make the enhanced features as close as possible to the true value label provided by the data set. With With As close as possible.

Images