A Method and System for Infrared and Visible Image Fusion Based on Global Integration of Multi-Scale Information and Preservation of Salient Regions

Abstract

This invention discloses a method and system for fusing infrared and visible light images based on global integration of multi-scale information and preservation of salient regions, relating to the field of infrared and visible light image fusion. The method includes: constructing a neural network based on global integration of multi-scale information and preservation of salient regions; constructing a loss function module based on saliency similarity, training the neural network to obtain a trained neural network; and using the trained neural network to obtain a fused image result of infrared and visible light images. This invention focuses on global integration based on multi-scale information, emphasizing the mining of global dependencies between features of multiple receptive fields. It specifically designs an encoder module for global integration of multi-scale information, a feature fusion module based on preservation of salient regions, and a saliency similarity loss function based on features of intermediate layers of the network, which can better improve the visual effect and fusion quality of the fused infrared and visible light images.

Description

Technical Field

[0001] This invention belongs to the field of image processing, specifically relating to an infrared and visible light image fusion method and system based on global integration of multi-scale information and preservation of salient regions. Background Technology

[0002] Infrared and visible light image fusion refers to the effective combination of information from infrared and visible light images to overcome their respective limitations and obtain a richer and more comprehensive image representation. Infrared and visible light image fusion is commonly used in low-light environments, nighttime surveillance, and adverse weather conditions, and has significant practical implications. Specifically, infrared images can provide substantial information about heat sources or targets in environments with insufficient visible light, while visible light images offer higher spatial resolution and texture detail. The fused image not only clearly displays infrared targets but also provides richer texture information, significantly improving overall visual quality and offering an effective approach for tasks such as target tracking, surveillance, and target detection.

[0003] Due to differences in lighting, weather, and environment in real-world imaging scenarios, or variations in the external and internal characteristics of the imaging target, infrared and visible light image fusion tasks often face problems of inconsistency and complementarity of image information. To overcome these issues, considering the differences in spatial resolution, texture information, and other features between infrared and visible light images, research on fusion algorithms has received widespread attention. Currently, deep learning-based infrared and visible light image fusion methods can be broadly categorized into convolutional neural network (CNN)-based algorithms, generative adversarial network (GAN)-based methods, and autoencoder-based methods. CNN-based methods perform poorly in the absence of real labels because their performance heavily depends on the loss function and network design. GAN-based methods provide a feasible solution for image fusion, but they face challenges such as high training complexity, instability during training, and low resource utilization efficiency. Autoencoder-based image fusion methods consist of an encoder, a fusion module, and a decoder, offering flexible fusion strategies, strong adaptability, and robust performance. However, existing autoencoder-based algorithms have limitations in feature extraction, potentially leading to the loss of some global information and insufficient preservation of scene location information.

[0004] The autoencoder algorithm, which combines convolutional neural networks and Transformers, can compensate for the lack of global information extraction by extracting features from the source image. However, its extraction method based on independent dual branches of the source image suffers from local information loss. Currently, there are few algorithms that leverage the advantages of the autoencoder algorithm that combines convolutional neural networks and Transformers by integrating multi-scale information globally. Summary of the Invention

[0005] To address the problems in the prior art, this invention proposes an infrared and visible light image fusion method and system based on global integration of multi-scale information and preservation of salient regions.

[0006] The technical solution adopted in this invention is as follows:

[0007] In a first aspect, the present invention discloses an infrared and visible light image fusion method based on global integration of multi-scale information and preservation of salient regions, comprising the following steps:

[0008] Step 1): Obtain a grayscale image that is registered and aligned with the infrared and visible light images, and obtain the blue and red density offset maps of the visible light image;

[0009] Step 2): Construct a neural network based on global integration of multi-scale information and preservation of salient regions. The neural network takes the registered and aligned infrared and visible light images as input and finally outputs the fused grayscale image. The neural network includes an encoder module for global integration of multi-scale information, a feature fusion module based on preservation of salient regions, and a feature decoder module.

[0010] Step 3): Construct a loss function based on saliency similarity, and train the neural network based on global integration of multi-scale information and preservation of salient regions to obtain a trained neural network;

[0011] Step 4): For the infrared and visible light images to be fused, a trained neural network is used to obtain the fused grayscale image, and an image post-processing module is constructed to perform image mode transformation on the fused grayscale image output by the neural network to obtain the fused color image; thus realizing the fusion of infrared and visible light images.

[0012] Secondly, the present invention discloses an infrared and visible light image fusion system based on multi-scale information global integration and salient region preservation, comprising:

[0013] The image preprocessing module is used to acquire a grayscale image that is registered and aligned with the infrared and visible light images, and to obtain a blue and red density offset map of the visible light image.

[0014] A neural network construction module is used to construct a neural network based on global integration of multi-scale information and preservation of salient regions. The neural network is used to obtain a fused image based on grayscale images of infrared and visible light images. The neural network includes an encoder module for global integration of multi-scale information, a feature fusion module based on preservation of salient regions, and a feature decoder module.

[0015] The image post-processing module is used to perform image mode transformation on the fused grayscale image to obtain the fused color image;

[0016] The neural network training module is used to construct a loss function module based on saliency similarity to train a neural network based on global integration of multi-scale information and preservation of salient regions, thereby obtaining a trained neural network.

[0017] The infrared and visible light image fusion module is used to obtain the fused image result of the infrared and visible light images to be fused using a trained neural network, thereby realizing the fusion of infrared and visible light images.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0019] 1) This invention proposes an encoder module based on global integration of multi-scale information, which effectively combines convolutional neural networks and Transformers to give full play to their respective advantages and extract multi-scale local features and global correlations at different levels.

[0020] 2) This invention proposes a fusion module based on salient region preservation. By using a spatial selection mechanism, the fusion weights are dynamically adjusted at the feature level to generate a salient attention map, thereby preserving the effective information of salient regions in the source image.

[0021] 3) This invention proposes a loss function based on saliency similarity, which adaptively calculates the global structural correlation of the image based on the extracted global features, and uses the attention map in the fusion module to calculate the similarity loss of salient regions, thus preserving effective information in important regions. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating the basic steps of an embodiment of the infrared and visible light image fusion method of the present invention;

[0023] Figure 2 This is a schematic diagram of the infrared and visible light image fusion device of the present invention;

[0024] Figure 3 This is a schematic diagram of the multi-scale sensing module of the present invention;

[0025] Figure 4 This is a schematic diagram of the feature fusion module based on salient region preservation according to the present invention;

[0026] Figure 5 The image shows the fusion results of the MSRS infrared and visible light image dataset fused using embodiments of the present invention and different methods. Detailed Implementation

[0027] The present invention will be further described and illustrated below with reference to specific embodiments. The embodiments described are merely examples of the content of this disclosure and do not limit the scope of the invention. The technical features of each embodiment in the present invention can be combined accordingly, provided that there is no mutual conflict.

[0028] like Figure 1 The diagram shown is a flowchart illustrating the basic steps of an embodiment of the infrared and visible light image fusion method of the present invention, mainly including:

[0029] Step 1): Image preprocessing

[0030] For the infrared and visible light images to be processed, this embodiment employs a method of mapping the RBG image mode to the YCbCr image mode, so that the visible light image in the RBG mode... To convert to YCbCr image format, where W represents the image width and H represents the image height, the calculation formula is as follows:

[0031] I VI-Y ,I VI-Cb ,I VI-Cr =RTY(I VI-RGB )

[0032] Among them, I VI-Y ,I VI-Cb ,I VI-Cr This represents the luminance component, blue and red density offset component map of a visible light image in YCbCr mode. RTY(·) represents the operation of mapping the image from RBG mode to YCbCr mode, and the specific calculation formula is as follows:

[0033] Y=0.299·R+0.587·G+0.114·B

[0034] Cb=-0.1687·R-0.3313·G+0.5·B

[0035] Cr=0.5·R-0.4600·G-0.0402·B

[0036] Wherein, Y, Cb, and Cr are the brightness components, blue and red concentration offset components of the image in YCbCr mode, respectively, and R, G, and B are the three color channel values ​​of the image in RBG mode; then, for the registered and aligned infrared and visible light images, the infrared image I... IR ∈R H×W×1 Brightness composition diagram I of visible light image VI-Y As input to the subsequent neural network.

[0037] Step 2): Construct a neural network based on global integration of multi-scale information and preservation of salient regions.

[0038] The neural network is used to obtain a fused image based on grayscale images of infrared and visible light images. The neural network includes an encoder module for global integration of multi-scale information, a feature fusion module based on salient region preservation, and a feature decoder module. The encoder module extracts multi-source features containing rich multi-scale and global correlation information from the source images. The feature fusion module obtains an adaptive salient region mask for the multi-source features and weights the multi-source features to obtain the final fused features. The feature decoder module processes the fused features and reconstructs the image to generate the fused image. This embodiment uses random parameters as the initial weights of the network.

[0039] The specific working process of each module in the neural network based on global integration of multi-scale information and preservation of salient regions is as follows:

[0040] Step 21): The encoder module for global integration of multi-scale information includes B serial multi-scale sensing modules and a Transformer-based global feature extraction module; the structure of the multi-scale sensing module is as follows: Figure 3 As shown, the multi-scale sensing module first bases its input preprocessed infrared image I... IR Brightness composition diagram I of visible light image VI-Y Multi-level, multi-scale features are constructed, and the calculation formula is as follows:

[0041] F (1) =MP(I)

[0042] F (i) =MP(F (i-1) )

[0043] Where MP(·) represents a multi-scale sensing module, I∈{I IR ,I VI-Y}, i = 1, 2, ..., B. In this embodiment, B = 5. The output of the (i-1)th multi-scale sensing module will continue to be fed into the ith multi-scale sensing module. The multi-level outputs are concatenated to obtain the final multi-scale information features, calculated using the following formula:

[0044] Φ L =Concat(F (1) ,F (2) ,…,F (B) )

[0045] in, It describes multi-level, multi-scale information features, and Concat(·) represents the channel splicing operation; This refers to the multi-level, multi-scale information features of infrared images. This refers to the multi-level, multi-scale information features of visible light images.

[0046] Then, for the extracted multi-level, multi-scale information features To further establish the global correlation between information at different scales, a global feature extraction module based on Transformer is constructed, the structure of which is as follows: Figure 1 As shown, firstly, multi-level, multi-scale information features are downsampled and embedded into the hidden layer to encode dense feature blocks, preparing for global context extraction. The Transformer-based global feature extraction module contains L Transformer layers, each consisting of a normalization layer, a multi-head attention mechanism, and a multilayer perceptron. After the encoded features pass through L Transformer layers, the features are progressively upsampled to recover the spatial dimension. The output features at different scales are concatenated with the features from the downsampling stage to output an infrared image feature tensor containing rich local context information. and visible light image feature tensor Therefore, the encoder outputs a total of 4 features:

[0047] Step 22): The features output by the encoder module include features with multi-scale information. Global features after integrating local information with global features This invention designs a feature fusion module based on salient region preservation, which generates a set of adaptive weights for each feature, and obtains the fused features based on the weights. Its structure is as follows: Figure 4 As shown. The feature fusion module based on salient region preservation first concatenates the four input features, and then applies average pooling and max pooling operations to the concatenated feature Φ, as expressed in the following formula:

[0048]

[0049] in, and These represent average pooling and max pooling operations, respectively. avg PM max These are their corresponding saliency feature descriptors. To facilitate the interaction of saliency information among different source features, the saliency feature descriptor PM... avg and PM max They are aggregated into a single salient feature descriptor. Then, they undergo a convolution operation. This is converted into N corresponding saliency attention maps, expressed as follows:

[0050]

[0051] in, This represents a saliency attention map; in this embodiment, N = 4. For each saliency attention map... The Sigmoid activation function is applied to obtain the corresponding saliency selection mask, which is used to extract features from infrared and visible light images. The calculation formula is as follows:

[0052]

[0053] in, Let ρ(·) represent the saliency selection mask, and let ρ(·) represent the Sigmoid activation function used to generate dynamic fusion weights. Then, features extracted from infrared and visible light images are weighted and fused using the saliency selection mask to obtain the fused feature tensor:

[0054]

[0055] The feature fusion module based on salient region preservation ultimately outputs fused features containing salient information from multiple source images.

[0056] Step 23): The feature decoder module aims to fully restore the fused features to the original size of the initial input image and output the fused grayscale image. The entire process is based on a multi-layer neural network, expressed by the formula:

[0057]

[0058] in, The fused grayscale image generated for the model, LeakyReLU and Softmax represent the Leaky ReLU nonlinear activation function and the Softmax nonlinear activation function, respectively, and Conv represents a two-dimensional convolutional layer.

[0059] Step 3: Training the Neural Network Model

[0060] A loss function based on saliency similarity is constructed to guide the neural network. This loss function comprises three components: gradient loss, mean squared error loss, and saliency similarity loss. By combining these loss functions, the model can learn local and global information, retain salient features, and maintain the structural integrity of the fused image. The formula for calculating the loss function is as follows:

[0061] L=λ1L grad +λ2L MSE +λ3L S-SIM

[0062] Among them, L grad L represents the gradient loss. MSE L represents the mean squared error loss. S-SIMThis represents the loss due to significant similarity. λ1, λ2, and λ3 are the balancing weights for each loss component. In this embodiment, λ1 = 50, λ2 =

[0063] 5, λ3 = 2.

[0064] The gradient loss function is used to preserve the details and texture information of the fused image. It calculates the gradient difference at corresponding locations by comparing the gradients of the fused image and the source image, squares the difference, and averages it over all pixels. The calculation formula is as follows:

[0065]

[0066] in, This represents the gradient of the image at pixel i, N is the total number of pixels in the image, and max indicates taking the larger value.

[0067] The mean squared error loss component evaluates the fusion effect by calculating the squared difference between the fused image and the source image at each pixel location. This loss helps to make the fused image as close as possible to the source image, and the calculation formula is as follows:

[0068]

[0069] Here, α1 and α2 are parameters used to balance the mean square error loss between the infrared image and the visible image. In this embodiment, α1 = α2 = 1.

[0070] The saliency similarity loss consists of two main components: saliency region similarity loss and global structural similarity loss. The expression is as follows:

[0071] L S-SIM =β1L SRS +β2L GSS

[0072] Among them, L SRS L represents the loss of significant region similarity. GSS This represents the global structural similarity loss. β1 and β2 are the corresponding weights. In this embodiment, β1 = 1 and β2 = 5.

[0073] The salient region similarity loss aims to preserve effective information about salient regions in an image. This is achieved by using a feature selection mask based on a salient region-preserving feature fusion module. The intensity information of the image is adaptively fused, and the calculation formula is as follows:

[0074]

[0075] Where δ represents the step function, and W1 and W2 are weights generated by selecting masks based on salient features of the visible and infrared images.

[0076] Global structural similarity loss is used to preserve the global structure of an image. Structural similarity is calculated by comparing the global features obtained from the Transformer-based feature extraction module. First, the mean μ of the global features is used... IR and μ VI The weights W3 and W4 for calculating global structural similarity are as follows:

[0077]

[0078] Here, MEAN(·) represents the mean operation. Based on the calculated weights W3 and W4, the structural similarity between the fused image and the source image is further calculated.

[0079] Finally, the global structural similarity loss is calculated using a weighted graph based on global features, as shown in the following formula:

[0080]

[0081] in, Infrared image I IR With the fused grayscale image I FUSE-Y Structural similarity between them Image I showing the brightness composition of a visible light image VI-Y With the fused grayscale image I FUSE-Y The structural similarity between them.

[0082] Using the entire infrared and visible light images as training samples, the network adopts an unsupervised training method. The samples are input into the network model, and the network weights are updated using the Adam gradient descent method with adaptive learning rate adjustment based on the constructed saliency similarity loss function module. In this embodiment, the training is iterated 60 times.

[0083] Step 4: Based on the trained neural network model, use the pre-processed infrared and visible light images as input to the network, and perform image post-processing on the output of the neural network based on global integration of multi-scale information and preservation of salient regions. The image post-processing steps are as follows:

[0084] For the fused grayscale image, this embodiment uses a method of mapping the YCbCr image mode to the RGB image mode, so that the fused grayscale image I in the YCbCr mode... Fuse-Y The formula for converting an image to RGB mode is as follows:

[0085] I FUSE =YTR(I FUSE-Y ,I VI-Cb ,I VI-Cr )

[0086] in, For the fused color image, I VI-Cb ,I VI-Cr This represents the blue and red density shift components of a visible light image in YCbCr mode. YTR(·) represents the method for mapping the image from YCbCr mode to RGB mode, and the specific calculation formula is as follows:

[0087] R = Y + 1.402·(Cr - 128)

[0088] G=Y-0.344136·(Cb-128)-0.714136·(Cr-128)

[0089] B = Y + 1.772 * (Cb - 128)

[0090] Finally, after passing through the image post-processing module, a three-channel color infrared and visible light fused image is obtained, realizing the result of infrared and visible light image fusion.

[0091] Figure 2 This is an example of an infrared and visible light image fusion system based on global integration of multi-scale information and preservation of salient regions, as illustrated in the embodiments. Figure 2 As shown, the system includes:

[0092] The image preprocessing module is used to acquire a grayscale image that is registered and aligned with the infrared and visible light images, and to obtain a blue and red density offset map of the visible light image.

[0093] A neural network construction module is used to construct a neural network based on global integration of multi-scale information and preservation of salient regions. The neural network is used to obtain a fused image based on grayscale images of infrared and visible light images. The neural network includes an encoder module for global integration of multi-scale information, a feature fusion module based on preservation of salient regions, and a feature decoder module.

[0094] The image post-processing module is used to perform image mode transformation on the fused grayscale image to obtain the fused color image;

[0095] The neural network training module is used to construct a loss function module based on saliency similarity to train a neural network based on global integration of multi-scale information and preservation of salient regions, thereby obtaining a trained neural network.

[0096] The infrared and visible light image fusion module is used to obtain the fused image result of the infrared and visible light images to be fused using a trained neural network, thereby realizing the fusion of infrared and visible light images.

[0097] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0098] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. For example, the image preprocessing module can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules can be combined or integrated into another unit. Furthermore, the connection between the displayed or discussed modules can be a communication connection through some interfaces, which can be electrical or other forms. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort. The following uses real infrared and visible light images as examples to illustrate specific implementation methods to demonstrate the technical effects of the present invention; specific steps in the embodiments will not be repeated.

[0099] Example

[0100] Next, using the MSRS infrared and visible light image dataset as the research object, we will verify the fusion method of this invention. To comprehensively compare the fusion performance, the fusion results are presented from both visual and quantitative perspectives, using fusion result images and evaluation metrics: mean square error (MSE) of the fused image, peak signal-to-noise ratio (PSNR), and artifact-based metric N. AB / F The proposed fusion method is evaluated using correlation coefficient (CC) and multi-scale structural similarity (MS-SSIM); the formulas are as follows:

[0101]

[0102] Among them, MSE X,F This represents the mean square error between the source image X and the fused image F, where A represents the visible light image and B represents the infrared image. A smaller mean square error value indicates that the difference between the fused image and the source image is smaller, meaning the fusion effect is better.

[0103] Peak signal-to-noise ratio (PSNR) quantifies the ratio of the maximum signal intensity to the noise level in the fused image. It indicates the degree of distortion introduced at the pixel level during the fusion process. The specific calculation formula is as follows:

[0104]

[0105] Here, r represents the peak value of the fused image. A larger peak signal-to-noise ratio (PSNR) indicates less distortion during the fusion process, resulting in better fusion quality.

[0106] NAB / F This metric measures the degree of artifacts introduced during the fusion process. The specific calculation formula is as follows:

[0107]

[0108] in, It measures the localization of artifacts in the fused image. Specifically, if the edge intensity at a certain location in the fused image is greater than the edge intensity at the corresponding location in the source image, that location is considered an artifact. and These represent the edge intensities of the fused image and the source image at position i, respectively. Lower N... AB / F The value indicates that fewer artifacts are introduced during the fusion process.

[0109] The correlation coefficient is used to evaluate the linear relationship between the fused image and the original image. The specific calculation formula is as follows:

[0110]

[0111] Where A represents a visible light image and B represents an infrared image. X i and F i Let i represent the pixel values ​​of the source image and the fused image at position i, respectively. and These represent the mean values ​​of the source image and the fused image, respectively. A higher CC value indicates a higher similarity between the fused image and the source image.

[0112] Multi-scale structural similarity, as a measure for analyzing image structure at multiple scales, can more comprehensively capture the structural features of an image. The calculation process for multi-scale structural similarity is as follows: First, the image is downsampled to obtain image pairs of different sizes. Then, the structural similarity value of each image pair at each scale is calculated. Finally, the structural similarity values ​​at different scales are weighted to obtain the multi-scale structural similarity. The specific calculation formula is as follows:

[0113] MS-SSIM X,F =1-MSSSIM(F,X)

[0114] MS-SSIM = MS-SSIM A,F +MS-SSIM B,F

[0115] Here, MSSSIM(·) represents multi-scale structural similarity calculation. A higher multi-scale structural similarity value indicates a higher structural consistency between the fused image and the source image.

[0116] The MSRS dataset contains 1444 infrared and visible light images, each containing 480×640 pixels.

[0117] Figure 5 The image shows the fusion results obtained from the MSRS dataset using embodiments of the present invention and different fusion algorithms.

[0118] Table 1 Evaluation metrics for the fusion results of MSRS infrared and visible light image datasets

[0119]

[0120] The comparative method MDLatLRR comes from: H.Li,

[0121] Comparative method FusionDN comes from: H.Xu, J.Ma, Z.Le, J.Jiang, and X.Guo, "Fusiondn: Aunified densely connected network for image fusion," Proceedings of the AAAIConference on Artificial Intelligence, vol.34, no.07, pp.12 484–12 491, 2020.

[0122] Comparative method ReCoNet comes from: Z.Huang, J.Liu, X.Fan, R.Liu, W.Zhong, and Z.Luo, "Reconet: Recurrent correction network for fast and efficient multi-modalityimage fusion," in Computer Vision–ECCV 2022, 2022, vol.13678, pp.539–555.

[0123] Introduction to SeAFusion: L. Tang, J. Yuan, and J. Ma,“Image fusion in theloop of high-level vision tasks:A semantic-aware real-time infrared andvisible image fusion network,”Information Fusion,vol.82,pp.28–42,2022.

[0124] Inventors of PIAFusion: L. Tang, J. Yuan, H. Zhang, X. Jiang, and J. Ma

[0125] “Piafusion:A progressive infrared and visible image fusion networkbased onillumination aware,”Information Fusion,vol.83–84,pp.79–92,2022.

[0126] Introduction to TarDAL: J. Liu, X. Fan, Z. Huang, G. Wu, R. Liu, W. Zhong, and Z. Luo, “Target-aware dual adversarial learning and a multi-scenario multi-modalitybenchmark to fuse infrared and visible for object detection,”in 2022IEEE / CVFConference on Computer Vision and Pattern Recognition(CVPR),2022,pp.5792–5801.

[0127] Introduction to SwinFusion: J. Ma, L. Tang, F. Fan, J. Huang, X. Mei, and Y. Ma, Swinfusion: Cross-domain long-range learning for general image fusion via swintransformer,” IEEE / CAA Journal of Automatica Sinica,vol.9,no.7,pp.1200–1217,2022.

[0128] Introduction to CDDFuse: Z. Zhao, H. Bai, J. Zhang, Y. Zhang, S. Xu, Z. Lin, R. Timofte, and L. Van Gool, “Cddfuse:Correlation-driven dual-branch featuredecomposition for multi-modality image fusion,”2023.

[0129] Introduction to NestFuse: H. Li, X.-J. Wu, and T. Durrani, “Nestfuse:An infraredand visible image fusion architecture based on nest connection and spatial / channel attention models,”IEEE Transactions on Instrumentation andMeasurement,vol.69,no.12,pp.9645–9656,2020.

[0130] Individual DIDFuse authors: Z.Zhao,S.Xu,C.Zhang,J.Liu,P.Li,and J.Zhang,

[0131] “Didfuse:Deep image decomposition for infrared and visible image fusion,” in Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, 2020, pp. 970–976.

[0132] Introduction to DeFusion: P. Liang, J. Jiang, X. Liu, and J. Ma, “Fusion fromdecomposition:A self-supervised decomposition approach for image fusion,”inComputer Vision–ECCV 2022,2022,vol.13678,pp.719–735.

[0133] Comparative method CrossFuse comes from: H.Li and

[0134] The fusion results of the MSRS datasets are as follows: Figure 5 As shown in Table 1, the quantization results demonstrate that this invention achieves the best performance, exhibiting good suppression of noise and artifacts present in the source image. This algorithm, based on neural network feature extraction with global integration of multi-scale information and preservation of salient regions, provides good representation of scene, location, and detailed texture when fusing images. Furthermore, the feature fusion module based on salient region preservation and the loss function module based on salient similarity effectively preserve richer fusion results.

[0135] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. Those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A method for fusing infrared and visible light images based on global integration of multi-scale information and preservation of salient regions, characterized in that, Includes the following steps: Step 1): Obtain a grayscale image that is registered and aligned with the infrared and visible light images, and obtain the blue and red density offset maps of the visible light image; Step 2): Construct a neural network based on global integration of multi-scale information and preservation of salient regions. The neural network takes the registered and aligned infrared and visible light images as input and finally outputs the fused grayscale image. The neural network includes an encoder module for global integration of multi-scale information, a feature fusion module based on preservation of salient regions, and a feature decoder module. The encoder module based on global integration of multi-scale information described in step 2) is used to mine multi-source features containing rich multi-scale information and global correlation information from the source image, wherein the source image is the brightness component map of infrared image and visible light image; the feature fusion module based on salient region preservation obtains an adaptive salient region mask for the multi-source features and weights the features to obtain the final fused features. The feature decoder module processes the fusion features and reconstructs the image to generate a fused image; The encoder module based on global integration of multi-scale information includes A serial multi-scale perception module and a Transformer-based global feature extraction module; The multi-scale sensing module first bases its input on a pre-processed infrared image. Brightness composition diagram of visible light image Multi-level, multi-scale features are constructed, and the calculation formula is as follows: ； ； in, This indicates a multi-scale sensing module. ;No. The output of the first multi-scale sensing module will continue to be fed into the second... A multi-scale sensing module; the multi-level outputs are concatenated to obtain the final multi-scale information features, calculated using the following formula: ； in, Describe multi-level, multi-scale information features. This indicates a channel splicing operation; This refers to the multi-level, multi-scale information features of infrared images. This refers to the multi-level, multi-scale information features of visible light images; Then, for the extracted multi-level, multi-scale information features and A global feature extraction module based on Transformer is constructed. First, multi-level, multi-scale information features are downsampled and embedded into hidden layers to encode dense feature blocks, preparing for global context extraction. The global feature extraction module based on Transformer includes... Each of the Transformer layers consists of a normalization layer, a multi-head attention mechanism, and a multilayer perceptron; when the encoded features are passed through... After several Transformer layers, the features are progressively upsampled to restore the spatial dimension. The output features at different scales are concatenated with the features from the downsampling stage to output an infrared image feature tensor containing rich local contextual information. and visible light image feature tensor Therefore, the encoder outputs a total of 4 features: ; Step 3): Construct a loss function based on saliency similarity, and train the neural network based on global integration of multi-scale information and preservation of salient regions to obtain a trained neural network; Step 4): For the infrared and visible light images to be fused, use the trained neural network to obtain the fused grayscale image, and construct an image post-processing module to perform image mode transformation on the fused grayscale image output by the neural network to obtain the fused color image; thus realizing the fusion of infrared and visible light images.

2. The infrared and visible light image fusion method based on global integration of multi-scale information and preservation of salient regions according to claim 1, characterized in that, Step 1) includes: Based on the input registered and aligned infrared and visible light images, the visible light image in RBG mode is... Convert to YCbCr image format. Indicates the width of the image. The height of the image is represented by the following formula: ； in, This represents the luminance component, blue density offset component, and red density offset component of a visible light image in YCbCr mode. This represents the operation of mapping an image from RBG mode to YCbCr mode; Then, for the registered and aligned infrared and visible light images, the infrared image... Brightness composition diagram of visible light images As input to the subsequent neural network.

3. The infrared and visible light image fusion method based on global integration of multi-scale information and preservation of salient regions as described in claim 1, characterized in that, The feature fusion module based on salient region preservation first processes the four input features... Perform splicing, and then analyze the spliced ​​features. The formulas for performing average pooling and max pooling operations are as follows: ； ； ； in, and These represent average pooling and max pooling operations, respectively. These are the corresponding saliency feature descriptors; to facilitate the interaction of saliency information between different source features, saliency feature descriptors... and They are aggregated into a single salient feature descriptor; then, they undergo a convolution operation. This is converted into four corresponding saliency attention maps, expressed as follows: ； in, This represents four saliency attention maps; for each saliency attention map... The Sigmoid activation function is applied to obtain the corresponding saliency selection mask, which is used to extract features from infrared and visible light images. The calculation formula is as follows: ； in, Indicates the saliency of the chosen mask. The Sigmoid activation function is used to generate dynamic fusion weights. Then, features extracted from infrared and visible light images are weighted and fused using a saliency-selective mask to obtain the fused feature tensor. ； The feature fusion module based on salient region preservation ultimately outputs fused features containing salient information from multiple source images. .

4. The infrared and visible light image fusion method based on global integration of multi-scale information and preservation of salient regions according to claim 3, characterized in that, The feature decoder module is based on a multi-layer neural network and outputs the fused grayscale image, expressed by the formula: ； in, The fused grayscale image generated for the model. and These represent the Leaky ReLU nonlinear activation function and the Softmax nonlinear activation function, respectively. This represents a two-dimensional convolutional layer.

5. The infrared and visible light image fusion method based on global integration of multi-scale information and preservation of salient regions according to claim 1, characterized in that, In step 4), Fusing grayscale images in YCbCr mode The formula for converting an image to RGB mode is as follows: ； in, For the merged color image, This represents the blue and red density shift composition maps of a visible light image in YCbCr mode. This represents the operation of mapping an image from YCbCr mode to RBG mode; Finally, after passing through the image post-processing module, a three-channel color infrared and visible light fused image is obtained.

6. The infrared and visible light image fusion method based on global integration of multi-scale information and preservation of salient regions according to claim 4, characterized in that, In step 3), The loss function based on saliency similarity It consists of three parts: gradient loss Mean square error loss and saliency similarity loss The formula for calculating the loss function is as follows: ； in, The balancing weights for each loss component; The formula for calculating the gradient loss is as follows: ； in, Indicates the image in pixels gradient at, It is the total number of pixels in the image. This indicates a larger value operation; The formula for calculating the mean squared error loss is as follows: ； in, and It is a parameter used to balance the mean square error loss between infrared and visible images; The saliency similarity loss includes saliency region similarity loss and global structural similarity loss; the expression is as follows: ； in, Indicates significant region similarity loss. Represents the global structural similarity loss. and These are the corresponding weights; The salient region similarity loss is achieved by using a feature selection mask based on a salient region-preserving feature fusion module. The intensity information of the image is adaptively fused, and the calculation formula is as follows: ； ； in, Represents the step function. and The weights for mask generation are selected based on salient features of visible and infrared images; Global structural similarity loss is used to preserve the global structure of an image. Structural similarity is calculated by comparing the global features obtained from the Transformer-based feature extraction module; firstly, the mean of the global features is used... and Calculate the weights of global structural similarity and The specific formula is as follows: ； ； in, This indicates a mean calculation operation; based on the calculated weights. and Further calculations are performed to determine the structural similarity between the fused image and the source image; Finally, the global structural similarity loss is calculated using a weighted graph based on global features, as shown in the following formula: ； in, Infrared image With the fused grayscale image Structural similarity between them A brightness composition diagram of a visible light image With the fused grayscale image Structural similarity between them; The loss function based on saliency similarity is used as the loss function, and the weights of the neural network are updated based on gradient descent with adaptive learning rate adjustment.

7. An infrared and visible light image fusion system based on global integration of multi-scale information and preservation of salient regions, implementing the method of claim 1, characterized in that, include: The image preprocessing module is used to acquire a grayscale image that is registered and aligned with the infrared and visible light images, and to obtain a blue and red density offset map of the visible light image. A neural network construction module is used to construct a neural network based on global integration of multi-scale information and preservation of salient regions. The neural network is used to obtain a fused image based on grayscale images of infrared and visible light images. The neural network includes an encoder module for global integration of multi-scale information, a feature fusion module based on preservation of salient regions, and a feature decoder module. The image post-processing module is used to perform image mode transformation on the fused grayscale image to obtain the fused color image; The neural network training module is used to construct a loss function module based on saliency similarity to train a neural network based on global integration of multi-scale information and preservation of salient regions, thereby obtaining a trained neural network. The infrared and visible light image fusion module is used to obtain the fused image result of the infrared and visible light images to be fused using a trained neural network, thereby realizing the fusion of infrared and visible light images.

Images