An image fusion method based on multi-scale strategy

Abstract

The application belongs to the field of image processing, and discloses an image fusion method based on a multi-scale strategy, which is used for solving the problems of poor feature extraction capability and poor fusion effect of a complex dual-mode image based on a deep learning-based image fusion method.The fusion method is based on a codec structure, and the core lies in that different feature extraction operations are performed on different scale features separated by an encoder, then a fusion module containing double-path processing is used to fuse the extracted multi-level features, and finally the fused features of different levels are spliced and up-sampled in the channel dimension to reconstruct a fused image.The application can effectively realize the fusion of a complex dual-mode image, has the characteristics of good multi-scale feature extraction effect, high target contrast of a fused image, high reconstruction precision, strong robustness and the like.

Description

Technical fields:

[0001] This invention relates to an image fusion method, specifically an image fusion method based on a multi-scale strategy. Background technology:

[0002] Images are among the most widely used types of data, but the information provided by a single-modal image sometimes fails to meet human visual needs or the requirements for target recognition and detection, especially when the difference between the target and background is small in a certain wavelength band. Therefore, it is necessary to use cameras with different imaging modes to capture images of different modalities, and to obtain a fused image with multi-modal information through image fusion. Image fusion technology uses algorithms to extract information from two or more images of the same scene from different sensors or with different focal points and brightness, integrating them into a single fused image to generate a fused image containing characteristics of all modalities. This comprehensively represents the scene, increases target contrast, and facilitates subsequent visual tasks. Compared to a single source image, a fused image has multi-modal information, higher target contrast, and more scene details. Currently, image fusion methods are gradually shifting from traditional, manually designed methods to deep learning-based methods. Traditional image fusion methods are mature, relying on manually designed image decomposition, feature fusion, and image reconstruction algorithms, which have high human error and computational costs, and perform poorly when dealing with complex images. Deep learning-based fusion methods delegate some or all of the work to neural networks, leveraging the powerful end-to-end fitting capabilities of neural networks to achieve automated updates of fusion parameters, thus better adapting to images of different modalities. However, current deep learning-based fusion methods still suffer from problems such as poor utilization of multi-scale information, inaccurate feature extraction, and low target contrast in the fusion results. Summary of the Invention:

[0003] The purpose of this invention is to overcome the shortcomings of the prior art and provide an image fusion method based on a multi-scale neural network structure. The image fusion method can achieve efficient utilization of multi-scale features, effectively fuse edge and target features of different scales, and reconstruct a fused image with more edge information and better target contrast.

[0004] The technical solution of this invention to solve the above-mentioned technical problems is: an image fusion method based on a multi-scale strategy, comprising the following steps:

[0005] (S1) Use cameras with different imaging modes to capture 2000 pairs of bimodal images to form a bimodal image set D to be fused. The images of the two modes are denoted as A1 and A2 respectively.

[0006] (S2) Using the bimodal images A1 and A2 as network inputs, the intermediate features F containing four scales are obtained through the encoder subnetwork.A1 and F A2 ;

[0007] (S3) Utilize the trained fusion subnetwork to fuse the intermediate features F of the two modalities. A1 and F A2 The fusion feature F is obtained. fused :

[0008] (S4) Fuse features F fused The input decoder subnetwork reconstructs the fused image F;

[0009] (S5) Construct the above sub-network into a multi-scale image fusion network, and calculate the loss function Loss value between the output fused image F and the input multimodal images A1 and A2. Train the multi-scale image fusion network by backpropagating the gradient of the Loss value until the Loss value converges, and obtain the trained multi-scale image fusion network.

[0010] (S6) The dual-modal image set D to be fused is put into the trained multi-scale image fusion network to obtain the fused image F.

[0011] Preferably, in step (S1), the dual-modal image includes, but is not limited to, visible light images, infrared images of different bands, linear polarization degree images (DoLP), etc.

[0012] Preferably, in step (S2), the encoder subnetwork includes four inverse residual convolutional modules (IR) and two position-aware convolutional modules (ParC). The IR module consists of a convolutional layer with a kernel size of 1×1, a depth convolutional layer with a kernel size of 3×3, a convolution with a kernel size of 1×1, and a nonlinear activation function ReLU, used to separate basic multi-scale features. The ParC module consists of position-aware convolutional blocks, global horizontal circular convolutions, global vertical circular convolutions, a feedforward neural network, and a channel attention module. The channel attention module includes a global average pooling module and two fully connected layers, used to extract deep-level global features.

[0013] Preferably, in step (S3), the fusion subnetwork consists of two paths: first, the intermediate feature F... A1 and F A2 The feature map of the i-th channel is extracted. One approach directly adds the two feature maps together, while the other approach concatenates the two feature maps by channel dimension, performs three 3×3 convolutions, and then adds them to the first approach to obtain the fused feature map of the i-th channel. After processing and stitching together all channels, the total fusion feature F is obtained. fused .

[0014] Preferably, in step (S4), the decoder subnetwork includes 6 decoding blocks (DCBs), each consisting of a 3×3 convolution kernel and a 1×1 convolution kernel. The decoding blocks are supplemented by differential upsampling and skip connections to achieve the fusion of multi-scale features.

[0015] Preferably, in step (S5), the loss function Loss is calculated by the following formula.

[0016] Loss=λ1L MWSSIM +λ2L ACOS (1)

[0017] Among them, L MWSSIM The multi-scale weighted structural similarity loss function is used to construct the background and texture loss between the input and fused images, and is calculated by equation (2). ACOS To adjust the cosine similarity loss function to constrain the target features of the fused image, it is calculated by equation (4), where λ1 and λ2 are hyperparameters used to balance the two loss functions.

[0018]

[0019] Where, β w L is the weighting coefficient. SSIM (x,y;w) represents the local structural similarity of images x and y within window w, calculated by equation (3).

[0020]

[0021] in σ(w) represents the mean of window w in image x. x w y ) for w x and w y The covariances, C1 and C2, are constants to prevent the divisor of the equation from being zero. L ACOS Calculated using the following formula:

[0022]

[0023] Among them, F fused It is a fusion feature, F en The intermediate features obtained by the encoder are fed into the output image.

[0024] The advantages of this invention compared to the prior art are as follows:

[0025] 1. The image fusion method based on multi-scale strategy of the present invention first uses the inverse residual module to extract features of four scales of the image. Then, different methods are used to further process the features of different scales. Since shallow features contain more texture and detail information, in order to preserve image details, shallow features are not processed too much. Deep features contain more global features of the target, so position-aware recurrent convolution is used to extract global features of deep features. This processing method can make targeted use of features of different scales to achieve better feature extraction performance, while avoiding excessive increase in network parameters.

[0026] 2. The image fusion method based on multi-scale strategy of the present invention adopts a dual-path parallel feature fusion approach. First, the feature maps of the i-th channel from different modal images are extracted. One path directly adds the feature maps to obtain a feature map that preserves texture details. The other path concatenates the two feature maps in the channel dimension and then performs three convolutions to obtain a feature map that fuses the overall features. Finally, the feature maps of the two paths are added to obtain the fused feature map of the i-th channel. This feature fusion method processes texture and target features separately, which can make full use of different components of texture and overall features at different scales, and achieve target feature fusion while preserving good texture information.

[0027] 3. The image fusion method based on multi-scale strategy of the present invention uses a weighted structural similarity loss function when constructing the loss function, which can constrain the brightness, contrast and structure of the image to be similar to the source image from different scales. At the same time, it uses the high-dimensional consistency of the adjusted cosine similarity loss function to constrain the multi-dimensional target features of the fused image to be consistent with the multi-dimensional target features of the source image. The combination of the two loss functions can enable the fused image to have both good texture and detail information and high target contrast. Attached image description:

[0028] Figure 1 This is a flowchart of the image fusion method based on a multi-scale strategy according to the present invention.

[0029] Figure 2 This is the overall network diagram of the image fusion method based on multi-scale strategy of the present invention.

[0030] Figure 3 This is a location-aware convolutional block diagram of the image fusion method based on a multi-scale strategy of the present invention.

[0031] Figure 4 This is a diagram of the residual fusion module of the image fusion method based on multi-scale strategy of the present invention.

[0032] Figure 5 This is a schematic diagram illustrating the training and specific testing execution process of the image fusion method based on multi-scale strategy of the present invention. Detailed implementation method:

[0033] The present invention will be further described in detail below with reference to embodiments, but the implementation of the present invention is not limited thereto.

[0034] See Figure 1 The image fusion method based on a multi-scale strategy of the present invention includes the following steps:

[0035] (S1) Use cameras with different imaging modes to capture 2000 pairs of bimodal images to form a bimodal image set D to be fused. The image resolution is 256×256. The images of the two modes are denoted as A1 and A2 respectively.

[0036] (S2) Using the bimodal images A1 and A2 as network inputs, the intermediate features F containing four scales are obtained through the encoder subnetwork. A1 and F A2 ;

[0037] (S3) Utilize the trained fusion subnetwork to fuse the intermediate features F of the two modalities. A1 and F A2 The fusion feature F is obtained. fused ;

[0038] (S4) Fuse features F fused The input decoder subnetwork reconstructs the fused image F;

[0039] (S5) Construct the above sub-network into a multi-scale image fusion network, and calculate the loss function Loss value between the output fused image F and the input multimodal images A1 and A2. Train the multi-scale image fusion network by backpropagating the gradient of the Loss value until the Loss value converges, and obtain the trained multi-scale image fusion network.

[0040] (S6) The dual-modal image set D to be fused is put into the trained multi-scale image fusion network to obtain the fused image F.

[0041] See Figure 2 The four IR blocks in the encoder subnetwork are inverse residual convolutional blocks, including a convolutional layer with a kernel size of 1×1, a depth convolutional layer with a kernel size of 3×3, a convolution with a kernel size of 1×1, and a nonlinear activation function ReLU. This module can effectively extract image features with only significant target features, such as infrared images and DoLP images, by increasing the number of channels through 1×1 convolution and then through 3×3 depth convolution. At the same time, the number of parameters is less than that of a regular convolution with a kernel size of 3×3. The final 1×1 convolution is used for channel information interaction and controlling the number of feature channels. Therefore, the IR module is suitable for separating the initial four scale features.

[0042] See Figure 2 and Figure 3 The position-aware recurrent convolutional block (ParC) used in the encoder subnetwork includes a position-aware convolutional block (PWConv), global horizontal circular convolution (GCC-H), global vertical circular convolution (GCC-V), a feedforward neural network (FFN), and a channel attention module (CWA). It assigns spatial weights through position-aware convolution, and then extracts deep global spatial features through continuous global horizontal and global vertical circular convolutions. After concatenating the features and performing regularization, channel features are extracted through the feedforward neural network and the channel attention module, thereby realizing the extraction of global spatial and channel features. It is suitable for extracting deep global features.

[0043] Since the features extracted by the encoder subnetwork contain four scales, and different scales contain different texture and target feature components, methods such as channel splicing, addition, or traditional manually designed fusion methods cannot adapt to or process the complex features with different components. It is necessary to process the texture details and target features separately in order to effectively preserve and fuse the corresponding features.

[0044] For this reason, see Figure 2 and Figure 4 The RF module in the fusion subnetwork contains two operations that can process texture details and target features separately: first, the intermediate features are processed by RF... A1 and F A2 The feature map of the i-th channel is extracted. One approach directly adds the two feature maps, preserving the texture information of the intermediate features. The other approach concatenates the two feature maps along their channel dimensions and then performs three 3×3 convolutions to fuse the target features. This fused feature map is then added to the first approach to obtain the fused feature of the i-th channel. After processing and stitching together all channels, the total fusion feature F is obtained. fused This fusion module achieves separate processing of edge and target features through two-way processing, avoiding the influence of a single processing method on the other feature.

[0045] The fused feature F fused The fusion features, which contain four scales, need to be merged and reconstructed into a fusion image with one channel. This requires continuous upsampling and stitching operations to merge information from each scale while reducing the number of channels.

[0046] For this reason, see Figure 2 The decoder subnetwork links six decoding blocks (DCBs) through interpolation upsampling and skip connections to achieve the merging of multi-scale features and channel dimensionality reduction. Each decoding block contains a convolutional layer with a kernel size of 3×3 and a convolutional layer with a kernel size of 1×1. Finally, the reconstruction of the fused image is achieved through a convolutional layer with a kernel size of 1×1.

[0047] See Figure 5 The loss function of the image fusion network based on a multi-scale strategy, L, is a multi-scale weighted structural similarity loss function. MWSSIM To construct the background and texture loss between the input and fused images, the adjusted cosine similarity loss function L is used. ACOS To constrain the target features of the fused image, L MWSSIM Calculated using the following formula:

[0048]

[0049] Where, β w The weighting coefficient, L, is calculated using equation (2). Its significance lies in the fact that images with larger variance contain more image information and therefore require larger weights. SSIM (x,y;w) represents the local structural similarity of images x and y within window w, calculated by equation (3).

[0050]

[0051] Where g(x) = max(x, 0.0001) is the correction coefficient used to improve stability, and σ 2 (·) indicates variance calculation. This represents the variance of image A1 within window w.

[0052]

[0053] in σ(w) represents the mean of window w in image x. x w y ) for wx and w y The covariances, C1 and C2, are constants of 0.0001 and 0.0009, respectively, used in the method equation where the divisor is 0. L ACOS Calculated using the following formula:

[0054]

[0055] Among them, F fused It is a fusion feature, F en The intermediate features obtained by the encoder are fed into the output image.

[0056] Finally, the total loss function of the network is:

[0057] Loss=λ1L MWSSIM +λ2L ACOS (5)

[0058] Where λ1 and λ2 are hyperparameters of 0.1 and 1, respectively.

[0059] The above are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above content. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. An image fusion method based on a multi-scale strategy, characterized in that, Includes the following steps: (S1) Use cameras with different imaging modes to capture 2000 pairs of bimodal images to form a bimodal image set D to be fused. The images of the two modes are denoted as A1 and A2 respectively. (S2) Using the bimodal images A1 and A2 as network inputs, the intermediate features F containing four scales are obtained through the encoder subnetwork. A1 and F A2 The encoder subnetwork includes four inverse residual convolutional modules (IR) and two position-aware convolutional modules (ParC). The IR module consists of a 1×1 convolutional layer, a 3×3 deep convolutional layer, a 1×1 convolutional layer, and a ReLU nonlinear activation function, used to separate basic multi-scale features. The ParC module consists of position-aware convolutional blocks, global horizontal circular convolutions, global vertical circular convolutions, a feedforward neural network, and a channel attention module. The channel attention module includes a global average pooling module and two fully connected layers, used to extract deep-level global features. (S3) Use the trained fusion subnetwork to fuse the intermediate features F of the two modalities. A1 and F A2 The fusion subnetwork consists of two paths: first, the intermediate features F... A1 and F A2 The feature map of the i-th channel is extracted. One approach directly adds the two feature maps together, while the other approach concatenates the two feature maps by channel dimension, performs three 3×3 convolutions, and then adds them to the first approach to obtain the fused feature map of the i-th channel. After processing and stitching together all channels, the fused feature F is obtained. fused ; (S4) Fuse feature F fused The input decoder subnetwork reconstructs the fused image F; (S5) Construct a multi-scale image fusion network by combining the encoder sub-network, fusion sub-network and decoder sub-network, and calculate the loss function Loss value between the output fused image F and the input multimodal images A1 and A2. Train the multi-scale image fusion network by backpropagating the gradient of the Loss value until the Loss value converges, and obtain the trained multi-scale image fusion network. (S6) The dual-modal image set D to be fused is put into the trained multi-scale image fusion network to obtain the fused image F.

2. The image fusion method based on a multi-scale strategy according to claim 1, characterized in that, In step (S1), the dual-modal image includes a visible light image, infrared images of different bands, and a linear polarization image.

3. The image fusion method based on a multi-scale strategy according to claim 1, characterized in that, In step (S4), the decoder subnetwork includes 6 decoding blocks (DCBs). Each decoding block consists of a 3×3 convolution kernel and a 1×1 convolution kernel. The decoding blocks are supplemented by differential upsampling and skip connections to achieve the fusion of multi-scale features.

4. The image fusion method based on a multi-scale strategy according to claim 1, characterized in that, In step (S5), the loss function Loss is calculated by the following formula: Among them, L MWSSIM The multi-scale weighted structural similarity loss function is used to construct the background and texture loss between the input and fused images, and is calculated by equation (2). ACOS To adjust the cosine similarity loss function to constrain the target features of the fused image, it is calculated by equation (4), where λ1 and λ2 are hyperparameters used to balance the two loss functions; Where, β w L is the weighting coefficient. SSIM (x, y; w) represents the local structural similarity of images x and y within window w, calculated by equation (3): in This represents the mean of window w in image x. For w x and w y The covariances, C1 and C2 are constants to prevent the divisor of the equation from being zero; L ACOS Calculated using the following formula: Among them, F fused It is a fusion feature, F en The intermediate features obtained by the encoder are fed into the output image.

Images