Multi-scene image fusion method

Abstract

The invention discloses a multi-scene image fusion method. According to the multi-scene image fusion method, image features under multiple scenes are continuously learned by establishing a multi-scale adaptive fusion network. The feature expression ability in different scenes is enhanced by fusing the local features of the CNN and the global features of the Transform. A new FFM structure is provided to fuse multi-scale and multi-scene features, and vision enhancement and image fusion technologies are coupled to improve the image fusion quality. A memory perception synaptic continuous learning method is adopted to train a model and calculate a loss function, visible light features of a fused image in different scenes are reserved, damage to color information under dark light is effectively reduced, and the problem of knowledge loss in the task switching process is relieved.

Description

Technical Field

[0001] The invention belongs to the field of image fusion, and in particular relates to a multi-scene image fusion method. Background Art

[0002] So far, many methods have been proposed for the fusion of infrared and visible light images. According to experimental results, the fusion effect based on deep learning methods is significantly better than that of traditional fusion methods. At present, deep learning methods can be mainly divided into three categories: image fusion models based on auto encoders (AE), fusion models based on generative adversarial networks (GAN), and image fusion models based on convolutional neural networks (CNN). The AE-based method uses auto encoders to complete feature extraction and feature reconstruction, and realizes image fusion through specific fusion strategies. The GAN-based method introduces the generative adversarial mechanism into the field of infrared and visible light image fusion, which mainly includes two parts: generator and discriminator. The discriminator is used to constrain the distribution of the fusion results generated by the generator so that it is as close to the source image as possible without supervision. The CNN-based method can realize feature extraction, feature fusion and feature reconstruction, and realize image fusion by designing the network architecture and loss function.

[0003] Although existing methods can effectively integrate important and complementary information from visible light and infrared images, there are still some problems to be overcome. Existing methods are designed for normal lighting conditions, thus ignoring the degradation of visible light images under insufficient lighting or low light conditions. Specifically, in low light conditions, previous fusion methods only use infrared images to fill in the scene defects caused by the degradation of visible light images. Therefore, the rich scene information in nighttime visible light images cannot be expressed in the fused image, which deviates from the original intention of the fusion goal of infrared and visible light images.

[0004] In view of this, the present invention is proposed. Summary of the invention

[0005] The technical problem to be solved by the present invention is to overcome the deficiencies of the prior art and provide a multi-scene image fusion method to solve the problems raised in the above-mentioned background technology.

[0006] In order to solve the above technical problems, the basic concept of the technical solution adopted by the present invention is:

[0007] A multi-scene image fusion method comprises the following steps:

[0008] Step (1): Obtain visible light images and infrared images and extract preliminary features from them. Then, use a convolutional neural network to extract local detail features, and then use Transformer to capture global structural features.

[0009] Step (2): Input the local and global features extracted in step (1) into the multi-scale feature fusion module (FFM), and fuse the features of different resolutions and scales through downsampling and convolution operations;

[0010] Step (3): In the feature extraction and fusion process of step (1) and step (2), the memory-aware synapse (MAS) continuous learning mechanism is adopted to dynamically adjust and protect the key parameters in the model through the MAS mechanism;

[0011] Step (4): input the multimodal features fused in step (2) into the fusion image reconstruction network, upsample layer by layer and reconstruct the final fusion image;

[0012] Step (5): During the network training process from step (1) to step (4), the accuracy of feature extraction, fusion and reconstruction is improved by jointly optimizing the gradient loss, pixel loss and MAS regularization term.

[0013] Optionally, the steps of using a convolutional neural network to extract local detail features and then using a Transformer to capture global structural features are:

[0014] Step S101, when using a convolutional neural network to extract the initial features of the visible light image and the infrared image, the convolution kernel size of the convolutional neural network is set to 3, the convolution step size is 1, and the convolution padding is 1 to extract the local features in the image. After the local features are extracted, the Transformer layer is used to capture the spatial correlation information between different pixels in the image through a multi-head self-attention mechanism. First, the number of channels of the input feature is tripled through the self-attention mechanism, and then the query (Q), key (K) and value (V) of the feature are calculated by splitting the channel. The expression is, Among them, P is a learnable parameter, and then the self-attention calculation formula MHSA(Q, K, V) = softmax(Q·K T )·V performs self-attention calculation, where K T represents the transposed matrix of K, softmax(Q·K T )·V is used to capture the spatial correlation information between different pixels in the image;

[0015] Step S102, after the global feature modeling, a multi-layer perceptron (MLP) is used to further perform nonlinear transformation processing on the extracted global features, and the expression is: Among them, MHSA(·) represents multi-head self-attention, LN(·) represents layer normalization, x represents the features of the input Transformer layer, and MLP(·) represents multi-layer perceptron;

[0016] Step S103, in the feature extraction process, each "feature extraction block" is composed of multiple CNN layers and Transformer layers. The first layer extracts preliminary features through CNN, and the subsequent layers use a CNN layer and multiple Transformer layers stacked in a way to mine multi-modal information at multiple granularities and adjust the size and resolution of the features through downsampling. The expression is:

[0017] where σ(·) represents the LeakyReLU activation function, C(·) represents the CNN layer, and T k (·) represents k consecutive Transformer layers. For the second layer, k is 2, and for the third and fourth layers, k is 3. ↓ represents a downsampling layer with a downsampling factor of 2.

[0018] Optionally, during the feature extraction process, the number of feature channels increases and the resolution decreases. The feature sizes obtained by the first, second, third, and fourth stage feature extraction are

[0019] Optionally, step (2) performs the following steps when receiving visible light and infrared image features of different sizes and channels:

[0020] Step S201: Fuse the visible light and infrared image features of different sizes and channels, gradually increase the number of channels and reduce the feature size to obtain richer scene semantic information. The feature extraction output of each layer is used as the input of the next layer for feature superposition processing. Its expression is F i =φ i (FFM(F0,F1,...,F i-1 ), where FFM(·) represents the multi-scale feature fusion module, F i represents the features obtained by the feature extraction block in the i-th stage, φ i (·) represents the feature extractor of the i-th stage

[0021] Step S202: In the process of fusion of features, a convolution layer C(·) with a convolution kernel size of 1 and a downsampling operation ↓ are used to adjust the number of channels and resolution of the features to be consistent so that the fusion process proceeds smoothly. The expression for feature size adjustment after downsampling is: Where C(·) represents a convolution layer with a convolution kernel of 1, ↓ represents downsampling, and {x1, x2, ..., x n} is the feature of the input multi-scale feature fusion module;

[0022] Step S203: Add an extra dimension to the features with the same resolution and the same number of channels obtained in the previous step, and then concatenate them along the extra dimension to obtain the concatenated features, whose expression is: f = concat(x′1, x′2, ..., x′ n ) , where f represents the features after concatenation along the extra dimension, x′ n It represents the features after adding extra dimensions. Then, the concatenated features are further processed by a three-dimensional convolution kernel with a convolution kernel of 1 to fully integrate the multi-scale features.

[0023] Optionally, in step (3), when the MAS mechanism is used to dynamically adjust and protect the key parameters in the model, first, in each task training, MAS calculates the new task loss L task (θ, D) and the regularized loss of the parameter changes of the old task are controlled, and the joint loss function is: Among them, L task (θ, D) represents the task loss of the new task, D represents the entire dataset, λ represents the regularization coefficient, which controls the balance between task loss and parameter preservation, and θ i represents the current value of the i-th parameter, θ i，old represents the value of the i-th parameter in the old task, and Ω i Indicates the importance of the i-th parameter in MAS calculation, and its expression is: Where D represents the entire data set, including all samples, N represents the number of samples in the data set, and f(x) represents the output of the network. It represents the gradient value of the output to the parameter. By calculating the gradient and importance of each parameter, the MAS mechanism dynamically adjusts the parameter update in the model so that the training of the new task will not have an excessive impact on the key parameters of the old task, and the learned knowledge can be effectively retained in the new task.

[0024] Optionally, the specific steps of reconstructing the final fused image are:

[0025] Step S401: Use a three-dimensional pooling layer to compress the extra dimension generated when splicing features to 1, and finally remove the extra dimension to obtain the final fused feature f, whose expression is: f = MaxPool 3 (C 3 (f)), where MaxPool 3 (·) represents the maximum pooling layer in three dimensions, C 3 (·) indicates a 3D convolutional layer with a convolution kernel of 1;

[0026] Step S402: After the fusion features obtained in step S401 are processed, the resolution of the features is gradually restored through upsampling operations to ensure that the resolution of the final fusion image is consistent with the original input image, and the features are further processed through convolution operations to ensure that they are suitable for the target output;

[0027] Step S403: Input the extracted multimodal features into the fusion image reconstruction network, and gradually generate a fusion image with the same resolution as the original image by using convolution and upsampling. The expression is: Where concat(·,·) means concatenating two features along the channel dimension. i represents the features obtained by the fusion image reconstruction network in the i-th stage, They represent the visible light features and infrared features obtained in the i-th stage of the feature extraction network respectively, and ↑ represents upsampling.

[0028] Optionally, in the fusion image reconstruction stage, the feature change process is opposite to that in the feature extraction stage. The feature sizes obtained by the fusion image reconstruction network in the first, second, third, and fourth stages are respectively The fourth stage is processed by two CNN layers. The feature size after processing by the first CNN layer is (16, h, w), and the image size after processing by the second CNN layer is (1, h, w).

[0029] Optionally, the steps to improve the accuracy of feature extraction, fusion and reconstruction by jointly optimizing gradient loss, pixel loss and MAS regularization term are as follows:

[0030] Step S501: During the training process of the fused image, a gradient loss function is defined to calculate the difference between the gradient of the fused image and the input image. The formula is as follows: L Gard (θ, D) = ||ΔI f -max(ΔI1, ΔI2)||1, where I1 represents the visible light image, I2 represents the infrared image, and I f represents the fused image, Δ represents the edge feature extraction of the image using the Sobel operator, max(·,·) represents the maximum of the two matrices element by element, and ||·||1 represents a norm;

[0031] Step S502: define a pixel loss function to make the fused image have a smooth pixel transition effect, then calculate the difference in pixel values ​​and optimize its smoothness. The calculation formula of pixel loss is: L Pix (θ, D) = || I f -max(I1, I2)||1, through the maximization operation of pixel values, the brightness and contrast of the fused image are kept consistent in the multimodal image;

[0032] Step S503: In the first stage of training, the gradient loss, pixel loss and structural similarity loss are integrated to make the network retain as much useful information as possible in scenes with normal brightness. The total loss in the first stage is L Stage1 (θ, D) = α1L Gard (θ, D)+α2L Pix (θ, D)+α3L ssim (θ, D), where α1, α2, α3 represent hyperparameters;

[0033] Step S504: For the second stage of training, the formula is modified for the fusion task in the night environment so that the model has a strong fusion ability in the night scene. The loss function is defined as: In order to prevent the network from forgetting the knowledge of old tasks during the second stage of training, MAS loss is introduced in the second stage to constrain the importance of parameters and define the total loss of the second stage as: L Stage2 (θ, D) = β1L Gard (θ, D)+β2L Pix (θ, D)+β3L ssim (θ, D)+β4L MAS (θ, D), where β1, β2, β3, β4 represent hyperparameters.

[0034] After adopting the above technical solution, the present invention has the following beneficial effects compared with the prior art. Of course, any product implementing the present invention does not necessarily need to achieve all the advantages described below at the same time:

[0035] By establishing a multi-scale adaptive fusion network, we can continuously learn image features in multiple scenes. By fusing the local features of CNN and the global features of Transformer, we can enhance the feature expression ability in different scenes. We propose a new FFM structure to fuse multi-scale and multi-scene features, and couple visual enhancement and image fusion technology to improve the image fusion quality. We use the memory-aware synaptic continuous learning method to train the model and calculate the loss function, retaining the visible light features of the fused images in different scenes, effectively reducing the damage to color information in dark light, and alleviating the problem of knowledge loss during task switching.

[0036] The specific implementation modes of the present invention are further described in detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is the overall framework diagram of the MixFusion model;

[0038] Figure 2 This is the framework diagram of the multi-scale feature fusion module;

[0039] Figure 3Comparison of visual effects of the present invention on the TNO dataset. (a) visible light image, (b) infrared image, (c) CAMF, (d) SuperFusion, (e) DATFuse, (f) DenseFuse, (g) EMMA, (h) FusionGAN, (i) GANMcC, (j) LRRNet, (k) U2Fusion;

[0040] Figure 4 This is a line chart comparing the indicators of the present invention on the TNO dataset, where the horizontal axis represents the image pair and the vertical axis represents the specific indicator value;

[0041] Figure 5 Comparison of visual effects of the present invention on the MSRS dataset. (a) visible light image, (b) infrared image, (c) CAMF, (d) SuperFusion, (e) DATFuse, (f) DenseFuse, (g) EMMA, (h) FusionGAN, (i) GANMcC, (j) LRRNet, (k) U2Fusion;

[0042] Figure 6 This is a line chart comparing the indicators of the present invention on the MSRS data set, where the horizontal axis represents the image pair and the vertical axis represents the specific indicator value;

[0043] Figure 7 Comparison of visual effects of the present invention on the LLVIP dataset. (a) Visible light image, (b) Infrared image, (c) CAMF, (d) SuperFusion, (e) DATFuse, (f) DenseFuse, (g) EMMA, (h) FusionGAN, (i) GANMcC, (j) LRRNet, (k) U2Fusion;

[0044] Figure 8 This is a line chart comparing the indicators of the present invention on the LLVIP dataset, where the horizontal axis represents the image pair and the vertical axis represents the specific indicator value;

[0045] Fig. 9 This is a visual comparison of the ablation experiments conducted by the present invention on the TNO dataset, where (a) is a visible light image, (b) is an infrared image, (c) the Transformer is removed, (d) the FFM is removed, and (e) the MAS is removed. DETAILED DESCRIPTION

[0046] See also Figure 1-9 As shown, in this embodiment, a multi-scene image fusion method is provided, including the following steps:

[0047] Step (1): Obtain visible light images and infrared images and extract preliminary features from them. Then, use a convolutional neural network to extract local detail features, and then use Transformer to capture global structural features.

[0048] Step (2): Input the local and global features extracted in step (1) into the multi-scale feature fusion module (FFM), and fuse the features of different resolutions and scales through downsampling and convolution operations;

[0049] Step (3): In the feature extraction and fusion process of step (1) and step (2), the memory-aware synapse (MAS) continuous learning mechanism is adopted to dynamically adjust and protect the key parameters in the model through the MAS mechanism;

[0050] Step (4): input the multimodal features fused in step (2) into the fusion image reconstruction network, upsample layer by layer and reconstruct the final fusion image;

[0051] Step (5): During the network training process from step (1) to step (4), the accuracy of feature extraction, fusion and reconstruction is improved by jointly optimizing the gradient loss, pixel loss and MAS regularization term.

[0052] In this embodiment, the steps of using a convolutional neural network (CNN) to extract local detail features and then using a Transformer to capture global structural features are as follows:

[0053] Step S101, when using a convolutional neural network (CNN) to extract the initial features of the visible light image and the infrared image, the convolution kernel size of the convolutional neural network is set to 3, the convolution step size is 1, and the convolution padding is 1 to extract the local features in the image. After the local features are extracted, the Transformer layer is used to capture the spatial correlation information between different pixels in the image through a multi-head self-attention mechanism. First, the number of channels of the input feature is tripled through the self-attention mechanism, and then the query (Q), key (K) and value (V) of the feature are calculated by splitting the channel. The expression is, Among them, P is a learnable parameter, and then the self-attention calculation formula MHSA(Q, K, V) = softmax(Q·K T )·V performs self-attention calculation, where K T represents the transposed matrix of K, softmax(Q·K T) V is used to capture the spatial correlation information between different pixels in the image;

[0054] Step S102, after the global feature modeling, a multi-layer perceptron (MLP) is used to further perform nonlinear transformation processing on the extracted global features, and the expression is: Among them, MHSA(·) represents multi-head self-attention, LN(·) represents layer normalization, x represents the feature of the input Transformer layer, and MLP(·) represents a multi-layer perceptron. Specifically, for the calculation of multi-head self-attention, the number of channels of the input feature is first tripled through a convolutional layer with a convolution kernel size of 1 and a stride of 1, and then it is split along the channel to obtain q, k, v for self-attention calculation.

[0055] Step S103, in the feature extraction process, each "feature extraction block" is composed of multiple CNN layers and Transformer layers. The first layer extracts preliminary features through CNN, and the subsequent layers use a CNN layer and multiple Transformer layers stacked in a way to mine multi-modal information at multiple granularities and adjust the size and resolution of the features through downsampling. The expression is: where σ(·) represents the LeakyReLU activation function, C(·) represents the CNN layer, and T k (·) represents k consecutive Transformer layers. For the second layer, k is 2, and for the third and fourth layers, k is 3. ↓ represents a downsampling layer with a downsampling factor of 2.

[0056] In this embodiment, during the feature extraction process, the number of feature channels increases continuously, and the resolution decreases continuously. The feature sizes obtained by the first, second, third, and fourth stage feature extraction are respectively

[0057] In this embodiment, step (2) performs the following steps when receiving visible light and infrared image features of different sizes and channels:

[0058] Step S201: Fuse the visible light and infrared image features of different sizes and channels, gradually increase the number of channels and reduce the feature size to obtain richer scene semantic information. The feature extraction output of each layer is used as the input of the next layer for feature superposition processing. Its expression is F i =φ i (FFM(F0,F1,...,F i-1 ), where FFM(·) represents the multi-scale feature fusion module, F i represents the features obtained by the feature extraction block in the i-th stage, φ i (·) represents the feature extractor of the i-th stage

[0059] Step S202: In the process of fusion of features, a convolution layer C(·) with a convolution kernel size of 1 and a downsampling operation ↓ are used to adjust the number of channels and resolution of the features to be consistent so that the fusion process proceeds smoothly. The expression for feature size adjustment after downsampling is: Where C(·) represents a convolution layer with a convolution kernel of 1, ↓ represents downsampling, and {x1, x2, ..., x n } It is the feature of the input multi-scale feature fusion module;

[0060] Step S203: Add an extra dimension to the features with the same resolution and the same number of channels obtained in the previous step, and then concatenate them along the extra dimension to obtain the concatenated features, whose expression is: f = concat(x′1, x′2, ..., x′ n ), where f represents the features after concatenation along the extra dimension, x′ n It represents the features after adding extra dimensions. Then, the concatenated features are further processed by a three-dimensional convolution kernel with a convolution kernel of 1 to fully integrate the multi-scale features.

[0061] In this embodiment, in step (3), when the key parameters in the model are dynamically adjusted and protected by the MAS mechanism, first, in each task training, MAS calculates the new task loss L task (θ, D) and the regularized loss of the parameter changes of the old task are controlled, and the joint loss function is: Among them, L task (θ, D) represents the task loss of the new task, D represents the entire dataset, λ represents the regularization coefficient, which controls the balance between task loss and parameter preservation, and θ i represents the current value of the i-th parameter, θ i，old represents the value of the i-th parameter in the old task, and Ω i Indicates the importance of the i-th parameter in MAS calculation, and its expression is: Where D represents the entire data set, including all samples, N represents the number of samples in the data set, and f(x) represents the output of the network. It represents the gradient value of the output to the parameter. By calculating the gradient and importance of each parameter, the MAS mechanism dynamically adjusts the parameter update in the model so that the training of the new task will not have an excessive impact on the key parameters of the old task, and the learned knowledge can be effectively retained in the new task.

[0062] In this embodiment, the specific steps of reconstructing the final fused image are:

[0063] Step S401: Use a three-dimensional pooling layer to compress the extra dimension generated when splicing features to 1, and finally remove the extra dimension to obtain the final fused feature f, whose expression is: f = MaxPool 3 (C 3 (f)), where MaxPool 3 (·) represents the maximum pooling layer in three dimensions, C3 (·) indicates a 3D convolutional layer with a convolution kernel of 1;

[0064] Step S402: After the fusion features obtained in step S401 are processed, the resolution of the features is gradually restored through upsampling operations to ensure that the resolution of the final fusion image is consistent with the original input image, and the features are further processed through convolution operations to ensure that they are suitable for the target output;

[0065] Step S403: Input the extracted multimodal features into the fusion image reconstruction network, and gradually generate a fusion image with the same resolution as the original image by using convolution and upsampling. The expression is: Where concat(·,·) means concatenating two features along the channel dimension. i represents the features obtained by the fusion image reconstruction network in the i-th stage, They represent the visible light features and infrared features obtained in the i-th stage of the feature extraction network respectively, and ↑ represents upsampling.

[0066] In this embodiment, in the fusion image reconstruction stage, the feature change process is opposite to that in the feature extraction stage. The feature sizes obtained by the fusion image reconstruction network in the first, second, third and fourth stages are respectively The fourth stage is processed by two CNN layers. The feature size after processing by the first CNN layer is (16, h, w), and the image size after processing by the second CNN layer is (1, h, w).

[0067] In this embodiment, the steps of improving the accuracy of feature extraction, fusion and reconstruction by jointly optimizing the gradient loss, pixel loss and MAS regularization term are as follows:

[0068] Step S501: During the training process of the fused image, a gradient loss function is defined to calculate the difference between the gradient of the fused image and the input image. The formula is as follows: L Gard (θ, D) = ||ΔI f -max(ΔI1, ΔI2)||1, where I1 represents the visible light image, I2 represents the infrared image, and I f represents the fused image, Δ represents the edge feature extraction of the image using the Sobel operator, max(·,·) represents the maximum of the two matrices element by element, and ||·||1 represents a norm;

[0069] Step S502: define a pixel loss function to make the fused image have a smooth pixel transition effect, then calculate the difference in pixel values ​​and optimize its smoothness. The calculation formula of pixel loss is: L Pix (θ, D) = || I f-max(I1, I2)||1, through the maximization operation of pixel values, the brightness and contrast of the fused image are kept consistent in the multimodal image;

[0070] Step S503: In the first stage of training, the gradient loss, pixel loss and structural similarity loss are integrated to make the network retain as much useful information as possible in scenes with normal brightness. The total loss in the first stage is L Stage1 (θ, D) = α1L Gard (θ, D)+α2L Pix (θ, D)+α3L ssim (θ, D), where α1, α2, α3 represent hyperparameters;

[0071] Step S504: For the second stage of training, the formula is modified for the fusion task in the night environment so that the model has a strong fusion ability in the night scene. The loss function is defined as: In order to prevent the network from forgetting the knowledge of old tasks during the second stage of training, MAS loss is introduced in the second stage to constrain the importance of parameters and define the total loss of the second stage as: L Stage2 (θ, D) = β1L Gard (θ, D)+β2L Pix (θ, D)+β3L ssim (θ, D)+β4L MAS (θ, D), where β1, β2, β3, β4 represent hyperparameters.

[0072] Experimental example: experimental verification and analysis

[0073] 1 Implementation Details

[0074] The proposed network code is implemented by Pytorch2.1.0 framework, Python version is 3.10, and Nvidia 3090 is used for training and testing in Ubuntu20.04. For the first stage of training, this paper selects all visible light images and infrared images from the TNO dataset, and obtains 3339 pairs of small blocks of size 128×128 by cropping for training the fusion network. During the training process, the initial learning rate is set to 0.001, the batch size is set to 16, and the training rounds are set to 20. When the training rounds are more than half, the learning rate of each round decays to 0.99 of the previous round. The parameters in the loss function are set to α1=50, α2=10, α3=10. When the gradient is generated, the Adam optimizer is used to update the network parameters.

[0075] For the second stage of training, we selected 200 pairs of low-light visible light images and infrared images from the MSRS dataset, expanded the data by cropping, and obtained 4895 pairs of small blocks of size 128 × 128. The training parameter settings are exactly the same as those in the first stage, and the parameters in the loss function are set to β1 = 50, β2 = 10, β3 = 10, and β4 = 1.

[0076] 2 Experimental setup

[0077] 2.1 Comparison of algorithms

[0078] In order to comprehensively analyze and compare the proposed methods, this paper selects the autoencoder-based model DenseFuse, the GAN-based models FusionGAN and GANMcC, the end-to-end CNN method-based models CAMF, LRRNet, U2Fusion, and the two-stage CNN methods EMMA and UMF-CMGR for quantitative and qualitative evaluation.

[0079] 2.2 Evaluation indicators

[0080] The evaluation of image fusion algorithms is mainly divided into qualitative evaluation of visual effects and quantitative evaluation of specific indicators. Image fusion tasks do not have specific true value labels, so the evaluation of visual effects is highly subjective. Generally speaking, it is mainly evaluated from several aspects such as brightness, clarity, the richness of texture details, the amount of observable effective information, and contrast.

[0081] The evaluation of visual effects cannot objectively compare the advantages and disadvantages of different fusion methods, so the comparison of quantitative indicators is needed to make a fair comparison of the effects of different methods. Common quantitative indicators include average gradient (AG), standard deviation (SD), information entropy (EN), visual fidelity (VIF), mutual information (MI), spatial frequency (SF), correlation sum of differences (SCD), etc. Average gradient (AG) represents the average value of the gradient in the fused image, which is used to objectively evaluate the richness of texture details in the fused image. The larger its value, the more edge texture details in the fused image. Spatial frequency (SF) is used to measure the frequency of pixel changes in the image. The larger the value, the clearer the image. Information entropy (EN) is an indicator based on information theory. The larger its value, the more information the fused image has. Standard deviation (SD) is used to reflect the intensity of the light and dark changes in the fused image. The higher its value, the higher the contrast of the obtained fused image. Visual fidelity (VIF) evaluates the quality of the fused image from the perspective of human visual imaging. The larger its value, the more the fused image conforms to human preferences. Mutual information (MI) is an important indicator used to measure the information integration ability of the fusion model. The larger its value, the stronger the information integration ability of the fusion model. The Qabf indicator plays a similar role to MI. The difference correlation sum (SCD) mainly measures the difference between the fused image and the input image. The larger its value, the greater the difference between the fused image and the source image, which proves that the generation effect of the model is better.

[0082] 2.3 Dataset

[0083] The TNO dataset is a standard test dataset widely used in infrared and visible light image fusion research. The dataset contains multiple pairs of registered infrared and visible light images and is usually used to evaluate and verify the performance of different image fusion algorithms. The scenes in the TNO dataset cover a variety of complex environments and conditions, such as cities, rural areas, transportation, buildings, night scenes, etc., ensuring the diversity of images under different lighting, weather and background complexity.

[0084] The MSRS (Multispectral Road Scenarios) dataset is a new infrared and visible light multispectral fusion dataset built on the basis of the MFNet dataset, with 1444 pairs of highly aligned infrared and visible light image pairs. After removing 125 pairs of unaligned image pairs from the MFNet dataset, the existing technology obtained 715 pairs of daytime images and 729 pairs of nighttime images.

[0085] The LLVIP (Low-Light Visible-Infrared Paired Dataset) dataset is a standard dataset designed specifically for visible light and infrared image fusion and target detection tasks under low-light conditions. This dataset provides paired visible light and infrared images, focusing on improving target perception and recognition in low-light environments, and is suitable for research in areas such as nighttime monitoring, autonomous driving, and security protection.

[0086] 3 Comparative experiment under normal lighting environment

[0087] In order to objectively compare the proposed method with similar methods, the comparison results of quantitative indicators on the TNO dataset are presented in Table 1, where the red data represent the best results. Thanks to the multi-scale feature fusion module of the proposed algorithm, the multi-modal features are fully fused, resulting in a significant lead in MI, Qabf and VIF indicators. U2Fusion brings rich edge detail information to the fused image through SSIM loss, resulting in the lead in SF and AG, so this paper retains this favorable factor during training. Considering that this loss may lead to a decrease in the ability to integrate information, this paper does not give it too much weight, and achieves the most balanced effect with the proposed gradient loss and pixel loss. SuperFusion can eliminate small displacements in multi-modal images during the fusion process, thereby achieving the best SSIM, but the other indicators are not leading. EMMA relies on a multi-stage fusion method to achieve the best EN, but the comprehensive indicators are weaker than the method proposed in this paper.

[0088] Table 1 Comparison of quantitative indicators on the TNO dataset

[0089]

[0090] In order to visually compare the proposed algorithm with similar algorithms, this paper selects two typical image pairs from the TNO dataset for comparison. Figure 3As shown in the figure, the red box marks the area with huge differences. For the first group of images, the visible light image can reflect the delicate background information, but the infrared image shows the opposite visual effect of the visible light image because the information of the two soldiers emitting thermal radiation is lost. The fused image obtained by CAMF can see both the background with rich details and the target emitting thermal radiation, but the information of the branches in the red box is not well preserved. The result of SuperFusion retains good detail information, the branches in the red box are clearer, but the thermal radiation information is not significant enough. The result of DATFuse is darker overall and the visual effect is average. The result of DenseFuse achieves worse scene brightness and loses all branch information. Thanks to the advantages of two-stage fusion, EMMA has good visual effects, significant infrared information, and the overall look and feel is almost the same as the method in this paper. The two GAN-based methods FusionGAN and GANMcC have blurred images and poor scene brightness. LrrNet ​​has good overall clarity, but it also has the problem of brightness degradation during the fusion process. U2Fusion also has poor scene brightness and serious loss of detail information. The method in this paper has high scene brightness, good visual effects, rich detail information in the red box, and significant thermal radiation information. For the second group of images, the hidden soldiers cannot be seen in the visible light image, and the detailed background information cannot be seen in the infrared image. The fusion result of CAMF has serious brightness degradation. The overall visual effect of SuperFusion is good, but the smoke information emitted from the chimney area is not significant enough. The fusion result of DATFuse is overall fuzzy. The result of DenseFuse has a brightness degradation problem. The visual effect of EMMA is similar to that of this article, but the chimney area is not as significant as the method in this article. The two GAN-based methods have serious scene brightness degradation. The infrared information of LRRNet is not significant enough, and U2Fusion has similar problems. The method in this article has high brightness, rich information, delicate images, and almost no obvious disadvantages.

[0091] In order to further compare the indicators of the proposed method and several comparative methods in detail, Figure 4 30 pairs of images were selected for point-to-point index comparison. The comparison results show that the proposed method has advantages in the comparison of different pairs of images, and the robustness and stability of the method have been verified.

[0092] 4. Comparative experiment in dark light environment

[0093] The method proposed in this paper uses the continuous learning theory to obtain fused images with good visual effects in both normal light and dark light environments. In this section, the fusion performance in dark light environments will be analyzed.

[0094] 4.1 Comparison on the MSRS dataset

[0095] In order to objectively compare the proposed method with similar algorithms, the results of quantitative comparison on the MSRS dataset are presented in Table 2, where the best indicators are marked in red. Due to the influence of gradient loss and FFM, the fused image texture obtained by the proposed method is clear, so it has a significant advantage in the comparison of single-source indicators such as EN, SF, AG, and VIF. Promoted by the MAS loss, the proposed method can adjust the brightness in a dark environment while retaining the strong information integration ability learned in the first stage, so it also has an advantage in the comparison of Qabf. EMMA relies on a two-stage fusion method, and its effect of information integration is better than that of the proposed method, so MI and SCD are ahead, but it is weaker than the proposed method in the comparison of comprehensive indicators.

[0096] Table 2 Quantitative comparison on the MSRS dataset

[0097]

[0098] In order to make an intuitive comparison, this paper selects two pairs of typical images for visual comparison. Figure 5 The comparison results are shown in . For the first set of images, the visible light image has a serious problem of overexposure of point light sources, and can only provide background grass information. The infrared image is stable, but the background details cannot be presented. CAMF handles the overexposed area well, but at the cost of lower overall brightness. SuperFusion does not handle the point light source problem well. DATFuse performs well on the MSRS dataset, but there is still a blur problem. DenseFuse scene brightness is seriously degraded. EMMA performs well on TNO, but on MSRS, facing some extreme environments, the effect is no longer excellent, and the point light sources in the image cannot be processed correctly. Both GAN-based methods and LrrNet ​​have the problem of image blur. U2Fusion infrared information is not significant, the scene brightness is seriously degraded, and the background information is almost completely lost. The scene brightness of this method is high, the brightness of the background area has not deteriorated, but has been improved, and the point light source is handled well, the details of the vehicle area are clear, and the texture is delicate. For the second set of images, the scene is no longer difficult, but the overall comparison shows similar results to the first set of images.

[0099] In order to further compare the indicators of the proposed method and several comparative methods in detail, Figure 6 30 pairs of images were selected for point-to-point index comparison. The comparison results once again show that the proposed method has advantages in the comparison of different pairs of images, and the robustness and stability of the method have been verified.

[0100] 4.2 Comparison on the LLVIP dataset

[0101] This paper selects the dark light dataset LLVIP to further verify the effectiveness of the proposed method. Table 3 shows the comparison of quantitative indicators on the LLVIP dataset. Similar to the results on the MSRS dataset, the proposed method achieves the best results in multiple indicators. DATFuse also adopts a hybrid structure of CNN and Transformer, so MI achieves the best, which also confirms the rationality of the sampling hybrid structure of this paper.

[0102] Table 3 Quantitative comparison on the LLVIP dataset

[0103]

[0104] In order to intuitively compare the proposed method with similar methods, Figure 7 Two groups of typical images were selected for visual comparison. For the first group of images, almost no pedestrian information can be seen in the visible light image, but the text information in the red marked box can be seen. The infrared image can clearly see the personnel information, but the text information is completely lost. The infrared information of CMAF is not significant enough. The infrared information of SuperFusion and DATFuse is more significant, but the text information is relatively fuzzy. The scene brightness of DenseFuse and EMMA is low, and the text information is lost. The two GAN-based methods have the problem of blurring again. LRRNet and U2Fusion not only have low scene brightness, but also the infrared information is not significant enough. The method in this paper has high scene brightness, significant infrared information, clear edges of the text information in the red marked box, and good overall visual effect. The second group of images further compares the significance of infrared information, and the conclusion is the same as the first group of images.

[0105] In order to further compare the indicators of the proposed method and several comparative methods in detail, Figure 8 30 pairs of images were selected for point-to-point index comparison. The comparison results once again show that the proposed method has advantages in the comparison of different pairs of images, and the robustness and stability of the method have been verified.

[0106] 5 Ablation Experiment

[0107] In order to verify the effectiveness of the proposed module, this paper conducted ablation experiments on Transformer, FFM and MAS losses. The experimental results are shown in Table 4. When the Transformer module is removed from the network, the feature expression ability of the network deteriorates, resulting in a decrease in the overall index. After removing FFM, the problem of insufficient feature fusion is brought about, which also leads to a decrease in performance. After removing MAS loss, the network will forget part of the knowledge learned in the first stage in the second stage of training, resulting in a slight decrease in the index.

[0108] Table 4 Quantitative comparison of ablation experiments

[0109]

[0110]

[0111] For intuitive comparison, Fig. 9 Two sets of images were selected from TNO for visual effect analysis. When the Transformer is removed, the decrease in feature expression ability leads to information loss, and the fused image becomes blurred as a whole. When the FFM is removed, the scale features cannot be fully integrated, which also causes blurring, but its negative impact on the network is less than the removal of the Transformer. After removing the MAS loss, the network forgets some knowledge in the second stage of training, resulting in the loss of information in some detail areas although the overall image effect is good.

[0112] In summary, the present invention achieves high-quality image fusion in different scenes. Specifically, the local features of CNN and the global features of Transformer are fused to enhance the feature expression capabilities in different scenes. Then, a novel FFM structure is proposed to fuse multi-scale and multi-scene features to couple visual enhancement and image fusion techniques to improve the image fusion quality. Finally, the MAS continuous learning method trains the model and calculates the loss function to retain the visible light features of the fused images in different scenes, effectively alleviating the problem of knowledge loss during task switching. The limitation of the method proposed in this paper is its poor ability to solve exposure degradation. In subsequent research, we will study image fusion technology for more extreme environments and more scenes.

[0113] The present invention is not limited to the above-mentioned embodiments. Anyone should be aware that any structural changes made under the enlightenment of the present invention, and any technical solutions that are the same or similar to the present invention, fall within the protection scope of the present invention. The technology, shape, and structural parts not described in detail in the present invention are all well-known technologies.

Claims

1. A multi-scene image fusion method, characterized in that: The following steps are involved: Step (1): Obtain visible light images and infrared images and extract preliminary features from them. Then, use a convolutional neural network to extract local detail features, and then use Transformer to capture global structural features. Step (2): Input the local and global features extracted in step (1) into the multi-scale feature fusion module (FFM), and fuse the features of different resolutions and scales through downsampling and convolution operations; Step (3): In the feature extraction and fusion process of step (1) and step (2), the memory-aware synaptic continuous learning mechanism is adopted to dynamically adjust and protect the key parameters in the model through the MAS mechanism; Step (4): input the multimodal features fused in step (2) into the fusion image reconstruction network, upsample layer by layer and reconstruct the final fusion image; Step (5): During the network training process from step (1) to step (4), the accuracy of feature extraction, fusion and reconstruction is improved by jointly optimizing the gradient loss, pixel loss and MAS regularization term.

2. The multi-scene image fusion method according to claim 1, characterized in that: The steps of using convolutional neural networks to extract local detail features and then using Transformer to capture global structural features are: Step S101: When the initial features of the visible light image and the infrared image are extracted using a convolutional neural network, the convolution kernel size of the convolutional neural network is set to 3, the convolution step size is 1, and the convolution padding is 1 to extract local features in the image. After the local features are extracted, the Transformer layer is used to capture the spatial correlation information between different pixels in the image through a multi-head self-attention mechanism. First, the number of channels of the input feature is tripled through the self-attention mechanism, and then the query (Q), key (K) and value (V) of the feature are calculated by splitting the channel. The expression is: Among them, P is a learnable parameter, and then the self-attention calculation formula MHSA(Q, K, V) = softmax(Q·K T )·V performs self-attention calculation, where K T represents the transposed matrix of K, softmax(Q·K T )·V is used to capture the spatial correlation information between different pixels in the image; Step S102: After the global feature modeling is completed, a multi-layer perceptron is used to further perform nonlinear transformation processing on the global features extracted above, and the expression is: Among them, MHSA(·) represents multi-head self-attention, LN(·) represents layer normalization, x represents the features of the input Transformer layer, and MLP(·) represents multi-layer perceptron; Step S103: In the feature extraction process, each feature extraction block is composed of multiple CNN layers and Transformer layers. The first layer extracts preliminary features through CNN, and the subsequent layers use a CNN layer and multiple Transformer layers stacked in a way to mine multi-modal information at multiple granularities and adjust the size and resolution of the features through downsampling. The expression is: where σ(·) represents the LeakyReLU activation function, C(·) represents the CNN layer, and T k (·) represents k consecutive Transformer layers. For the second layer, k is 2, and for the third and fourth layers, k is 3. ↓ represents a downsampling layer with a downsampling factor of 2.

3. The multi-scene image fusion method according to claim 2, characterized in that: In the process of feature extraction, the number of feature channels increases and the resolution decreases. The feature sizes obtained by the first, second, third, and fourth stage feature extraction are 4. The multi-scene image fusion method according to claim 1, characterized in that: Step (2) performs the following steps when receiving visible light and infrared image features of different sizes and channels: Step S201: Fuse the visible light and infrared image features of different sizes and channels, gradually increase the number of channels and reduce the feature size to obtain richer scene semantic information. The feature extraction output of each layer is used as the input of the next layer for feature superposition processing. Its expression is F i =φ i (FFM(F0,F1,...,F i-1 ), where FFM(·) represents the multi-scale feature fusion module, F i represents the features obtained by the feature extraction block in the i-th stage, φ i (·) represents the feature extractor of the i-th stage Step S202: In the process of fusion of features, a convolution layer C(·) with a convolution kernel size of 1 and a downsampling operation ↓ are used to adjust the number of channels and resolution of the features to be consistent so that the fusion process proceeds smoothly. The expression for feature size adjustment after downsampling is: Among them, C(·) represents a convolution layer with a convolution kernel of 1, ↓ represents downsampling, and {x1, x2, …, x n } is the feature of the input multi-scale feature fusion module; Step S203: Add an extra dimension to the features with the same resolution and the same number of channels obtained in the previous step, and then concatenate them along the extra dimension to obtain the concatenated features, whose expression is: f = concat(x′1, x′2, ..., x′ n ), where f represents the features after concatenation along the extra dimension, x′ n It represents the features after adding extra dimensions, and then the concatenated features are further processed by a three-dimensional convolution kernel with a convolution kernel of 1 to fuse the multi-scale features.

5. The multi-scene image fusion method according to claim 1, characterized in that: In step (3), when the MAS mechanism is used to dynamically adjust and protect the key parameters in the model, first, in each task training, MAS calculates the new task loss L task (θ, D) and the regularized loss of the parameter changes of the old task are controlled, and the joint loss function is: Among them, L task (θ, D) represents the task loss of the new task, D represents the entire dataset, λ represents the regularization coefficient, which controls the balance between task loss and parameter preservation, and θ i represents the current value of the i-th parameter, θ i，old represents the value of the i-th parameter in the old task, and Ω i Indicates the importance of the i-th parameter in MAS calculation, and its expression is: Where D represents the entire data set, including all samples, N represents the number of samples in the data set, and f(x) represents the output of the network. It represents the gradient value of the output to the parameter. By calculating the gradient and importance of each parameter, the MAS mechanism dynamically adjusts the parameter update in the model so that the training of the new task will not have an excessive impact on the key parameters of the old task, and the learned knowledge can be effectively retained in the new task.

6. The multi-scene image fusion method according to claim 1, characterized in that: The specific steps to reconstruct the final fused image are: Step S401: Use a three-dimensional pooling layer to compress the extra dimension generated when splicing features to 1, and finally remove the extra dimension to obtain the final fused feature f, whose expression is: f = MaxPool 3 (C 3 (f)), where MaxPool 3 (·) represents the maximum pooling layer in three dimensions, C 3 (·) indicates a 3D convolutional layer with a convolution kernel of 1; Step S402: After the fusion features obtained in step S401 are processed, the resolution of the features is gradually restored through upsampling operations to ensure that the resolution of the final fusion image is consistent with the original input image, and the features are further processed through convolution operations to ensure that they are suitable for the target output; Step S403: Input the extracted multimodal features into the fusion image reconstruction network, and gradually generate a fusion image with the same resolution as the original image by using convolution and upsampling. The expression is: Where concat(·,·) means concatenating two features along the channel dimension. i represents the features obtained by the fusion image reconstruction network in the i-th stage, They represent the visible light features and infrared features obtained in the i-th stage of the feature extraction network, and represent upsampling.

7. The multi-scene image fusion method according to claim 6, characterized in that: In the fusion image reconstruction stage, the feature change process is opposite to that in the feature extraction stage. The feature sizes obtained by the fusion image reconstruction network in the first, second, third, and fourth stages are The fourth stage is processed by two CNN layers. The feature size after the first CNN layer is (16, h, w), and the image size after the second CNN layer is (1, h, w).

8. The multi-scene image fusion method according to claim 1, characterized in that: Through the joint optimization of gradient loss, pixel loss and MAS regularization term, the steps to improve the accuracy of feature extraction, fusion and reconstruction are as follows: Step S501: During the training process of the fused image, a gradient loss function is defined to calculate the difference between the gradient of the fused image and the input image. The expression is: L Gard (θ, D) = |ΔI f -max(ΔI1, ΔI2)||1, where I1 represents the visible light image, I2 represents the infrared image, and I f represents the fused image, Δ represents the edge feature extraction of the image using the Sobel operator, max(·,·) represents the maximum of the two matrices element by element, and ||·||1 represents a norm; Step S502: define a pixel loss function to make the fused image have a smooth pixel transition effect, then calculate the difference in pixel values ​​and optimize its smoothness. The calculation formula of pixel loss is: Pix (θ, D) = || I f -max(I1, I2)||1, through the maximization operation of pixel values, the brightness and contrast of the fused image are kept consistent in the multimodal image; Step S503: In the first stage of training, the gradient loss, pixel loss and structural similarity loss are integrated to enable the network to retain useful information in scenes with normal brightness. The total loss in the first stage is L Stage1 (θ, D) = α1L Gard (θ,D)+α2L Pix (θ, D)+α3L ssim (θ, D), where α1, α2, α3 represent hyperparameters; Step S504: For the second stage of training, the formula is modified for the fusion task in the night environment so that the model has fusion capability in the night scene. The loss function is defined as: In order to prevent the network from forgetting the knowledge of old tasks during the second stage of training, MAS loss is introduced in the second stage to constrain the importance of parameters. The total loss of the second stage is defined as: L Stage2 (θ, D)=β1L Gard (θ, D)+β2L Pix (θ, D)+β3L ssim (θ, D)+β4L MAS (θ, D), among which β1, β2, β3, β4 are shown as parameters.

Images