A dual-decoder-based image fusion method

Abstract

The application belongs to the field of image fusion, and discloses a kind of image fusion method based on double decoder, for solving the problem that the feature extraction capability and fusion effect of complex multi-modal image processing of camera shooting of different imaging modes based on deep learning-based image fusion method are poor, the application includes: multi-modal image A1, A2 is extracted through large receptive field feature extraction module, then respectively through two interactive decoding modules, and in the decoding process, the decoding information between two different modal decoding modules is spliced on the channel to interact and fuse, finally reconstructs the fusion image C, calculates the loss of fusion image C and multi-modal image A1, A2, updates network model parameters.The application can effectively realize the fusion of complex multi-modal image, has the characteristics of good feature information extraction, small parameter quantity, high reconstruction precision, strong robustness and the like.

Description

Technical fields:

[0001] This invention relates to an image fusion method, specifically an image fusion method based on a dual decoder. Background technology:

[0002] With the advancement of technology, the information provided by a single source image cannot meet the needs of human vision or the requirements for target recognition and detection. Therefore, it is necessary to use cameras with different imaging modes to capture multimodal images and obtain fused images with richer details through image fusion.

[0003] Image fusion technology integrates information from two or more images of the same scene from different sensors or at different locations, times, and brightness levels into a single fused image. This comprehensive representation of the imaging scene facilitates subsequent visual tasks. Compared to a single source image, a fused image provides clearer information about the scene in which the target is located, significantly improving image quality and sharpness.

[0004] Traditional image fusion methods are relatively mature, but they require the manual design of complex fusion rules, resulting in high manual and computational costs. For complex multimodal images, they heavily rely on manually designed features, making it difficult to design universal feature extraction methods. In recent years, with the rise of deep learning, deep learning-based image fusion methods have emerged, providing new ideas for image fusion. However, current deep learning-based image fusion methods have high network complexity and high computational cost, and for complex multimodal images, they may still suffer from inaccurate feature extraction and poor image fusion results. Summary of the Invention:

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide an image fusion method based on simple gate unit non-activation feature extraction. The image fusion method can fuse complex multimodal images and has the characteristics of good feature information extraction, few parameters, high reconstruction accuracy, and strong robustness.

[0006] The technical solution of the present invention to solve the above-mentioned technical problems is:

[0007] An image fusion method based on dual decoders includes the following steps:

[0008] (S1) Use cameras with different imaging modes to capture multimodal images, denoted as images A1 and A2;

[0009] (S2) The multimodal images A1 and A2 are used as the input of the network. They are first passed through a convolutional layer and then through N large receptive field feature extraction modules to obtain the multimodal feature map.

[0010] (S3) The two multimodal feature maps are passed through two interactive decoding modules respectively. During the decoding process, the decoding information between the two different modal decoding modules is spliced ​​on the channel for interactive fusion. This is repeated N times to gradually fuse the images. Then, the images are passed through a convolutional layer to obtain the fused image C.

[0011] (S4) Construct a neural network using the above process, calculate the loss function value between the fused image output by the neural network and the input image, and backpropagate the gradient of the loss function value to update the parameters of the network model until the loss function value converges. Then stop updating the network model parameters to obtain a trained neural network.

[0012] Preferably, in step (S1), the multimodal image includes, but is not limited to, visible light image, short-wave infrared image, mid-wave infrared image, long-wave infrared image, and polarization image.

[0013] Preferably, in step (S1), the multimodal image A1 is a visible light image, and A2 is one of short-wave, medium-wave, long-wave infrared images or polarized images.

[0014] Preferably, in step (S2), the range of possible number of times N is repeated by the module is 4≤N≤6.

[0015] Preferably, in step (S2), the large receptive field feature extraction module adopts residual connection, including a convolutional layer with a kernel size of 1*1, a Gaussian error nonlinear activation function, a deep convolutional layer with a kernel size of 5*5, a deep convolutional layer with a kernel size of 5*5 and a dilation value of 3, and pixel normalization.

[0016] Preferably, in step (S3), the interactive decoding module acquires the feature information of each level and performs pixel superposition with the fusion decoding information of the previous level for interactive decoding.

[0017] Preferably, in step (S3), the interactive decoding module includes a convolutional layer with a kernel size of 3*3, channel attention, and interpolation upsampling.

[0018] Preferably, in step (S4), the loss function of the neural network is a loss function used to compare the similarity between the fused result image and the image before fusion. The loss function Loss is a combination of SSIM loss, background content loss, and salient target loss, and the expression of the loss function is as follows:

[0019] L SSIM =1-kSSIM(A1,C)-(1-k)SSIM(A2,C) (1)

[0020]

[0021]

[0022] Loss=δ1L SSIM +δ2L back +δ3L salient (4)

[0023] In the above formula δ1 is the gradient operator, h and w are the height and width of the image, respectively, k can take different values ​​for different modal images of the input, the value range of k is 0 < k < 1, and δ1 + δ2 + δ3 should be equal to 1.

[0024] Compared with the prior art, the present invention has the following advantages:

[0025] 1. The image fusion method based on dual decoders of this invention employs a large receptive field feature extraction module, using depthwise separable large kernel convolution to reduce the model size while increasing the receptive field. Using a large convolution kernel allows information to be collected from a large region, and enhances the extraction capability of semantic information. The designed depthwise separable large kernel convolution reduces the significant computational burden associated with large kernel convolution, thereby reducing parameters and achieving better feature extraction performance.

[0026] 2. The image fusion method based on dual decoders of this invention uses dual decoders and performs multimodal fusion during the decoding stage. In existing methods, multimodal fusion is typically performed during the encoding stage, but this fusion strategy is more difficult to optimize compared to methods that perform fusion during the decoding stage. During backpropagation, the gradient calculation path in the decoder is shorter than that in the encoder, therefore the optimization process of the decoder is less affected by gradient vanishing or exploding. Thus, the decoder is easier to optimize than the encoder.

[0027] 3. The dual-decoder-based image fusion method of this invention employs an interactive decoding module, aiming to leverage the complementary potential of multiple types of information from different modes and image content, including modal fusion information and contextual information. Unlike existing works that use different information individually to improve decoding features, we use the interactive decoding module in the dual-decoder as the basic module of the decoder, combining multiple types of information. Then, channel attention is used to adaptively select useful information, and feature reconstruction is performed after pixel summation of different types of information. Furthermore, our network also features interaction between the two decoders, whose output information includes fusion information and modal-specific information. This information is added to each decoding step, i.e., passed to the two interactive decoding modules at the next level, progressively improving the decoded feature information, thereby further enhancing the quality of multimodal image fusion. Attached image description:

[0028] Figure 1 This is a flowchart of the image fusion method based on dual decoders of the present invention.

[0029] Figure 2 This is a simplified structural diagram of the large receptive field feature extraction module in the dual-decoder-based image fusion method of the present invention.

[0030] Figure 3 This is a structural diagram of the interactive decoding module of the image fusion method based on dual decoders of the present invention. Detailed implementation method:

[0031] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0032] See Figures 1-3 The image fusion method based on dual decoders of the present invention includes the following steps:

[0033] (S1) Use cameras with different imaging modes to capture multimodal images, denoted as images A1 and A2;

[0034] (S2) The multimodal images A1 and A2 are used as the input of the network. They are first passed through a convolutional layer and then through N large receptive field feature extraction modules to obtain the multimodal feature map.

[0035] (S3) The two multimodal feature maps are passed through two interactive decoding modules respectively. During the decoding process, the decoding information between the two different modal decoding modules is spliced ​​on the channel for interactive fusion. This is repeated N times to gradually fuse the images. Then, the images are passed through a convolutional layer to obtain the fused image C.

[0036] (S4) Construct a neural network using the above process, calculate the loss function value between the fused image output by the neural network and the input image, and backpropagate the gradient of the loss function value to update the parameters of the network model until the loss function value converges. Then stop updating the network model parameters to obtain a trained neural network.

[0037] See Figure 2 In step (S2), the large receptive field feature extraction module adopts residual connection, including a convolutional layer with a kernel size of 1*1, a Gaussian error nonlinear activation function, a deep convolutional layer with a kernel size of 5*5, a deep convolutional layer with a kernel size of 5*5 and a dilation value of 3, and pixel normalization.

[0038] See Figure 3 In step (S3), the interactive decoding module acquires the feature information of each level and performs pixel superposition with the fusion decoding information of the previous level for interactive decoding.

[0039] See Figure 3In step (S3), the interactive decoding module includes a convolutional layer with a kernel size of 3*3, channel attention, and interpolation upsampling.

[0040] Furthermore, in this embodiment, the loss function of the neural network is a loss function used to compare the similarity between the fused image and the original image. The loss function Loss is a combination of SSIM loss, background content loss, and salient target loss, and its expression is as follows:

[0041] L SSIM =1-kSSIM(A1,C)-(1-k)SSIM(A2,C) (1)

[0042]

[0043]

[0044] Loss=δ1L SSIM +δ2L back +δ3L salient (4)

[0045] In the above formula δ1 is the gradient operator, h and w are the height and width of the image, respectively, k can take different values ​​for different modal images of the input, the value range of k is 0 < k < 1, and δ1 + δ2 + δ3 should be equal to 1.

[0046] In addition, the multimodal images described in this embodiment include, but are not limited to, visible light images, short-wave infrared images, mid-wave infrared images, long-wave infrared images, and polarized images.

[0047] In addition, in this embodiment, the multimodal image A1 is a visible light image, and A2 is a mid-wave or long-wave infrared image with an image resolution of 640*512.

[0048] In addition, the number of times N is repeated for the module described in this embodiment can be 4.

[0049] The above are merely preferred embodiments of the present invention, and the scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A dual-decoder based image fusion method, characterized in that, Includes the following steps: (S1) Use cameras with different imaging modes to capture multimodal images, denoted as images A1 and A2; (S2) The multimodal images A1 and A2 are used as the input of the network. They are first passed through a convolutional layer and then through N large receptive field feature extraction modules to obtain multimodal feature maps. The large receptive field feature extraction modules adopt residual connections and include a convolutional layer with a kernel size of 1*1, a Gaussian error nonlinear activation function, a deep convolutional layer with a kernel size of 5*5, a deep convolutional layer with a kernel size of 5*5 and a dilation value of 3, and pixel normalization. (S3) The two multimodal feature maps are passed through two interactive decoding modules, each of which includes a 3*3 convolutional layer, channel attention, and interpolation upsampling. Each interactive decoding module acquires feature information from each level and performs pixel stacking with the fused decoding information from the previous level for interactive decoding. During the decoding process, the decoding information between the two different modal interactive decoding modules is spliced ​​on the channels for interactive fusion, and this process is repeated N times to gradually fuse the images. The images are then passed through a convolutional layer to obtain the fused image C. (S4) Construct a neural network using the feature extraction and decoding processes described in steps (S2) and (S3), calculate the loss function value between the fused image output by the neural network and the input image, and backpropagate the gradient of the loss function value to update the parameters of the network model until the loss function value converges. Then stop updating the network model parameters to obtain a trained neural network.

2. The dual-decoder-based image fusion method according to claim 1, characterized in that, In step (S1), the multimodal image includes a visible light image, a short-wave infrared image, a mid-wave infrared image, a long-wave infrared image, and a polarization image.

3. The dual-decoder-based image fusion method of claim 1, wherein, In step (S1), the multimodal image A1 is a visible light image, and A2 is one of short-wave, medium-wave, long-wave infrared images or polarized images.

4. The dual-decoder-based image fusion method of claim 1, wherein, In step (S2), the possible range of the number of times N is repeated by the module is 4≤N≤6.

5. The image fusion method based on dual decoders according to claim 1, characterized in that, In step (S4), the loss function of the neural network is a loss function that compares the similarity between the fused image and the original image. The loss function Loss is a combination of SSIM loss, background content loss, and salient target loss. The expression of the loss function is as follows: L SSIM = 1 - kSSIM(A1, C) - (1 - k)SSIM(A2, C) Loss = δ1L SSIM + δ2L back + δ3L salient In the above formula is the gradient operator, h and w are the height and width of the image respectively, k can take different values for different modality images of the input, the value range of k is 0 < k < 1, and δ1 + δ2 + δ3 should satisfy being equal to 1.

Images