Infrared and visible light image fusion method based on conditional diffusion model

Abstract

The invention discloses an infrared and visible light image fusion method based on a conditional diffusion model, and belongs to the technical field of image fusion. Compared with a current popular infrared and visible light image fusion algorithm, the algorithm is obviously improved in the aspects of color fidelity, texture detail maintenance and fusion quality, and feature extraction and fusion tasks are completed through combination of a conditional denoising diffusion implicit model and a multi-channel high and low frequency parallel fusion model; the problem that the denoising diffusion implicit model lacks a reference true value of a fusion result when executing an image fusion task is relieved, the unique advantages of the denoising diffusion implicit model in image processing are further shown, and the method is excellent in the aspects of high-quality image generation and mathematical interpretability. The optimization of the existing method and the upgrading of the existing technology are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of image fusion, and in particular to an infrared and visible light image fusion method based on a conditional diffusion model. Background Art

[0002] Infrared and visible light image fusion can better fuse features from multiple types of sensors to generate high-quality fused images with comprehensive scene information. Infrared images can capture thermal radiation information and have significant contrast. Visible light images contain rich detail information and have high resolution, but single-modal image information is relatively simple and has its own shortcomings. In order to make the fusion result more consistent with human visual perception and provide more accurate semantic representation, infrared and visible light image fusion is widely used in remote sensing, all-weather detection, military reconnaissance and other fields. The fused image combines the thermal radiation information in the infrared image and the texture details of the visible light image, overcoming the limitations of visible light images being easily affected by the environment and infrared images lacking texture information.

[0003] The purpose of infrared and visible light image fusion is to use the trained network model for feature extraction without manually formulating artificial levels and measurement rules. By selecting appropriate fusion strategies, the advantages of different imaging principle information can be brought into play, the deficiencies of one imaging principle information can be made up, and the accuracy, comprehensiveness and usability of the obtained information can be improved, providing more effective and comprehensive information for all-weather detection and other tasks.

[0004] The core of improving infrared and visible light image fusion technology lies in optimizing algorithms and improving computing power. At present, deep learning has become the mainstream algorithm in the field of image fusion due to its powerful feature extraction capabilities. Deep learning algorithms mostly constrain fused images by constructing objective functions, and rely on the powerful nonlinear fitting ability of neural networks to obtain the desired distribution characteristics. Among them, the image fusion framework based on the GAN network has produced good fusion results, but there are still problems such as unstable training process and lack of interpretability. In recent years, due to the stability and controllability of the image generation process, the research on denoising diffusion probability models has received widespread attention.

[0005] Deep learning algorithms based on diffusion models have great advantages in image fusion and perform well in generating high-quality images and mathematical interpretability. However, there are still major challenges. Due to the lack of a benchmark true value for the fusion result, it is difficult to directly implement the image fusion task. To solve this problem, some methods use pre-trained diffusion models to provide prior knowledge and manually construct true values ​​to achieve image fusion. However, the pre-trained diffusion weights used are usually not suitable for the current image fusion task, resulting in insufficient feature extraction, loss of detailed information of the fused image, and poor quality. In addition, the structure of the pre-trained network limits the flexibility and application scope of the model. In order to improve the fusion quality, a conditional denoising diffusion model can be used to solve the problem of lack of true value fusion images.

[0006] The present invention provides an infrared and visible light image fusion method based on a conditional diffusion model. While solving the problem of lack of reference true values, it can significantly enhance the texture details and edge contour information of the fused image, perform well in visual effects and color fidelity, and improve its application in military, all-weather monitoring, medical technology and other fields. Summary of the invention

[0007] The purpose of the present invention is to solve the above-mentioned problems and to provide a method for fusion of infrared and visible light images based on a conditional diffusion model.

[0008] To achieve the above purpose, the technical solution adopted by the present invention is: the specific implementation steps are as follows;

[0009] 1) The registered single-channel infrared image and three-channel visible light image are spliced ​​into a four-channel image and input into the conditional diffusion module. The prior distribution of the custom subset is approximated as the prior distribution suitable for the infrared and visible light image feature extraction task. In the forward diffusion process of the conditional diffusion module, the model gradually adds Gaussian noise in the form of a Markov chain within T steps. The multi-channel noisy image Ft is expressed as:

[0010] (1)

[0011] in, represents the standard normal distribution, and Respectively represent the multi-channel noisy image and the original image of the current step length, 1- represents the product of all noise variances before step size t, express Obey Gaussian distribution.

[0012] In the reverse denoising process of the conditional diffusion module, the model gradually removes noise through T time steps. At time step t, the multi-channel source image The prediction of , which can be expressed as:

[0013] (2)

[0014] Furthermore, in the case of known multi-channel noisy images and a multi-channel source image When , the multi-channel noisy image of the previous time step can be derived , which can be expressed as:

[0015] (3)

[0016] in, represents the noise variance at time step t, Represents a noisy image The mean of the conditional distribution of can be expressed as:

[0017] (4)

[0018] in, represents the denoising network, and They represent the input noisy multi-channel image and time step of the denoising network respectively;

[0019] 2) Using the highest a priori distribution learned by the conditional diffusion model, combined with the denoising generation process and the multi-channel likelihood correction network, the multi-channel feature distribution of the spliced ​​source image is extracted; in the reverse denoising generation process, a latent space is set to introduce a multi-channel likelihood correction network. At each time step, the multi-channel source image Prediction The spliced ​​four-channel image is combined with the multi-channel EM algorithm for likelihood correction. The semantic information is gradually introduced in the iterative process, so that the denoising result can better retain the characteristic information of the source image. The diffusion modeling in the latent space is used to achieve accurate simulation of the multi-channel distribution of infrared and visible light images. The calculation process can be expressed as:

[0020] (5)

[0021] in, Represents a multi-channel source image The difference between the prediction and the visible light signature, Represents a multi-channel source image The difference between the prediction and the infrared signature, and represents the latent space scale parameter, represents the penalty term;

[0022] 3) The generation results of three different time steps in the denoising generation process of the conditional diffusion model are intercepted and spliced ​​to obtain multi-stage splicing features;

[0023] 4) The multi-stage infrared and visible light features generated by the denoising generation process are used through a detail adaptive denoising network to extract four-channel image features and separate high- and low-frequency features. First, the spliced ​​image of each time step is used as the input image, and a 3×3 convolution kernel is used to perform preliminary feature extraction and channel number alignment on the input image. Then, a deep neural network encoder is used to extract hierarchical and abstract features and align the number of channels. Using the hierarchical feature map generated by the encoder, the spatial and channel attention mechanism is applied to each layer of the feature map , which can be expressed as:

[0024] (6)

[0025] in, and Respectively represent the encoder Feature maps before and after layer processing, and Respectively represent the encoder Parameters of layer channels and spatial attention, Represents a convolution with a kernel size of 3, which is used to enhance the aggregation ability of spatial and channel attention.

[0026] Finally, the feature map extracted by the encoder is fed into the decoder, and the semantic and detail enhancement modules are used to connect the encoder and decoder stages at each level; , select the corresponding feature map For reference, the size of feature maps at different levels is adjusted using up and down sampling to fit the encoder feature map The resolution is calculated and element-by-element multiplication is performed. The center frequency domain filtering is performed in the frequency domain using Fourier transform to selectively filter out the low-frequency components in the feature map and retain the multi-channel high-frequency diffusion feature output. The calculation process can be expressed as:

[0027] (7)

[0028] (8)

[0029] (9)

[0030] in, , and Represent the adaptive average pooling, unit mapping and nearest neighbor upsampling operations respectively, and Denotes the decoder Feature maps before and after layer processing, and Respectively The height and width of represents element-wise multiplication, applied to all feature maps of adjusted resolution, and denote Fourier transform and inverse Fourier transform respectively, is the Fourier mask, is the center filter radius, is the threshold frequency;

[0031] 5) In the denoising network decoder stage, the current level feature map is passed Calculate the average value of the backbone feature map to achieve balanced adjustment of the extracted features; To avoid the extracted high-frequency features from producing too smooth textures, use the full-channel scaling operation only in the first layer of the current level, and limit the subsequent level scaling operations to The half channel retains the multi-channel low-frequency diffusion characteristic output. The calculation process can be expressed as:

[0032] (10)

[0033] (11)

[0034] in, and Represents feature maps The number of channels and the current decoder level, express No. channels;

[0035] 6) Output the image diffusion features extracted by the detail adaptive denoising network. The decoder of the detail adaptive denoising network contains 5 convolutional layer levels. Each level uses a 1×1 convolution kernel to output the multi-channel high-frequency diffusion features and multi-channel low-frequency diffusion features extracted at the current stage. The map corresponding to the high-frequency features is called the detail map, and the map corresponding to the low-frequency features is called the basic map;

[0036] 7) The multi-channel high-frequency diffusion features extracted by the detail adaptive denoising network are fused through the adaptive regional consistency fusion network. First, a symmetric filling operation is applied to the extracted multi-channel high-frequency diffusion features, a one-pixel border is added, and a 3×3 window is used for maximum filtering. The maximum filter can be expressed as:

[0037] (12)

[0038] in, represents the output image, Indicates The pixel values ​​of the 3×3 matrix centered at Indicates taking the maximum value.

[0039] Then, the sign function is applied to the multi-channel high-frequency diffusion features after maximization to generate an activity map. A 3×3 convolution layer is used to map the high-dimensional fusion features to the 128-channel output. The threshold of each pixel is calculated according to the local area of ​​the output, and the mapping matrix is ​​generated according to the adaptive threshold. The calculation process can be expressed as:

[0040] (13)

[0041] (14)

[0042] in, represents the symbolic function, and represents different filtered images, represents the mapping value of the convolution matrix, Represents the mapping matrix.

[0043] Finally, replace the parameter 0 in the mapping matrix with the pixel corresponding to the infrared image, and replace the parameter 1 with the pixel corresponding to the visible light image, and then map it through the convolution layer to obtain the fused detail image. ;

[0044] 8) The multi-channel low-frequency diffusion features extracted by the detail adaptive denoising network are fused through the multi-channel low-frequency feature fusion network. These multi-channel low-frequency features are first processed by combining the nearest neighbor upsampling and 1×1 convolution kernel, and then a series of convolution layers with a convolution kernel of 3×3 are used to map the low-dimensional information features to 3-channel outputs to obtain the fusion base map;

[0045] 9) The Concat strategy is used to fuse the fusion detail map dominated by high-frequency features obtained by the adaptive regional consistency fusion network and the fusion basic map dominated by low-frequency features obtained by the multi-channel low-frequency feature fusion network to reconstruct the fusion image of infrared and visible light.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] The key algorithm of the technical solution of the present invention is the conditional diffusion model fusion algorithm; compared with the currently popular infrared and visible light image fusion algorithm, the algorithm has obvious improvements in color fidelity, texture detail preservation and fusion quality. It not only completes the feature extraction and fusion tasks by combining the conditional denoising diffusion implicit model and the multi-channel high and low frequency parallel fusion model, alleviating the problem of the lack of a benchmark true value of the fusion result when the denoising diffusion implicit model performs image fusion tasks, but also further demonstrates the unique advantages of the denoising diffusion implicit model in image processing, and still performs well in generating high-quality images and mathematical interpretability, realizing the optimization of existing methods and the upgrading of existing technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a flow chart of the infrared and visible light image fusion method of the conditional diffusion model of the present invention;

[0049] Figure 2 A multi-channel likelihood correction module of the present invention;

[0050] Figure 3 A detail adaptive denoising network of the present invention;

[0051] Figure 4 The semantics and detail enhancement module of the present invention;

[0052] Figure 5 The adaptive regional consistency fusion network of the present invention;

[0053] Figure 6 It is a multi-channel low-frequency feature fusion network of the present invention;

[0054] Figure 7 This is the night scene fusion result of the MSRS dataset “00004N” of the present invention;

[0055] Figure 8 This is the daytime scene fusion result of the MSRS dataset “00241D” of the present invention;

[0056] Fig. 9 This is the scene fusion result of the RoadScene dataset “FLIR_0579” of the present invention. DETAILED DESCRIPTION

[0057] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below in conjunction with specific implementation methods.

[0058] The process of the infrared and visible light image fusion method of the technical solution of the present invention is as follows Figure 1 As shown:

[0059] The specific implementation steps are as follows;

[0060] 1) The registered single-channel infrared image and three-channel visible light image are spliced ​​into a four-channel image input into the conditional diffusion module, and the prior distribution of the custom subset is approximated to the prior distribution suitable for the infrared and visible light image feature extraction task, so that the denoising generation process can be more specifically designed according to the characteristics of the task, making the diffusion process more compact and more suitable for the feature extraction task of infrared and visible light image fusion. In the forward diffusion process of the conditional diffusion module, the model gradually adds Gaussian noise in the form of a Markov chain within T steps, and the multi-channel noisy image Ft is expressed as:

[0061] (1)

[0062] in, represents the standard normal distribution, and Respectively represent the multi-channel noisy image and the original image of the current step length, 1- represents the product of all noise variances before step size t, express Obey Gaussian distribution.

[0063] In the reverse denoising process of the conditional diffusion module, the model gradually removes noise through T time steps. At time step t, the multi-channel source image The prediction of , which can be expressed as:

[0064] (2)

[0065] Furthermore, in the case of known multi-channel noisy images and a multi-channel source image When , the multi-channel noisy image of the previous time step can be derived , which can be expressed as:

[0066] (3)

[0067] in, represents the noise variance at time step t, Represents a noisy image The mean of the conditional distribution of can be expressed as:

[0068] (4)

[0069] in, represents the denoising network, and They represent the input noisy multi-channel image and time step of the denoising network respectively.

[0070] 2) Using the highest a priori distribution learned by the conditional diffusion model, combined with the denoising generation process and the multi-channel likelihood correction network, the multi-channel feature distribution of the spliced ​​source image is extracted. In the reverse denoising generation process, the latent space is set to introduce the multi-channel likelihood correction network. The network architecture is as follows: Figure 2 As shown. At each time step, for the multi-channel source image Prediction The spliced ​​four-channel image is combined with the multi-channel EM algorithm for likelihood correction, and semantic information is gradually introduced in the iterative process, so that the denoising result can better retain the characteristic information of the source image. Through diffusion modeling in the latent space, the accurate simulation of the multi-channel distribution of infrared and visible light images is achieved. The calculation process can be expressed as:

[0071] (5)

[0072] in, Represents a multi-channel source image The difference between the prediction and the visible light signature, Represents a multi-channel source image The difference between the prediction and the infrared signature, and represents the latent space scale parameter, Represents a penalty term.

[0073] 3) The generation results of three different time steps in the denoising generation process of the conditional diffusion model are intercepted and spliced ​​to obtain multi-stage splicing features.

[0074] 4) The multi-stage infrared and visible light features generated by the denoising process are extracted through the detail adaptive denoising network to extract four-channel image features and separate high- and low-frequency features. The overall architecture of the detail adaptive denoising network is as follows: Figure 3 As shown in the figure. First, the spliced ​​image of each time step is used as the input image, and a 3×3 convolution kernel is used to perform preliminary feature extraction and channel number alignment on the input image. Then, a deep neural network encoder is used to extract hierarchical and abstract features, and the number of channels is aligned. The hierarchical feature map generated by the encoder is used to apply the spatial and channel attention mechanism to each layer of the feature map. , which can be expressed as:

[0075] (6)

[0076] in, and Respectively represent the encoder Feature maps before and after layer processing, and Respectively represent the encoder Parameters of layer channels and spatial attention, Represents a convolution with a kernel size of 3, which is used to enhance the aggregation ability of spatial and channel attention.

[0077] Finally, the feature map extracted by the encoder is fed into the decoder, and the semantic and detail enhancement module is used to connect the encoder and decoder stages of each level. The architecture of the semantic and detail enhancement module is as follows: Figure 4 For different levels of encoder , select the corresponding feature map For reference, the size of feature maps at different levels is adjusted using up and down sampling to fit the encoder feature map The resolution is calculated and element-by-element multiplication is performed. The center frequency domain filtering is performed in the frequency domain using Fourier transform to selectively filter out the low-frequency components in the feature map and retain the multi-channel high-frequency diffusion feature output. The calculation process can be expressed as:

[0078] (7)

[0079] (8)

[0080] (9)

[0081] in, , and Represent the adaptive average pooling, unit mapping and nearest neighbor upsampling operations respectively, and Decoder Feature maps before and after layer processing, and Respectively The height and width of represents element-wise multiplication, applied to all feature maps of adjusted resolution, and denote Fourier transform and inverse Fourier transform respectively, is the Fourier mask, is the center filter radius, is the threshold frequency.

[0082] 5) In the denoising network decoder stage, the current level feature map is passed Calculate the average value of the backbone feature map to achieve a balanced adjustment of the extracted features. To avoid the extracted high-frequency features from producing too smooth textures, use the full-channel scaling operation only in the first layer of the current level, and limit the subsequent level scaling operations to The half channel retains the multi-channel low-frequency diffusion characteristic output. The calculation process can be expressed as:

[0083] (10)

[0084] (11)

[0085] in, and Represents feature maps The number of channels and the current decoder level, express No. channels.

[0086] 6) Output the image diffusion features extracted by the detail adaptive denoising network. The decoder of the detail adaptive denoising network contains 5 convolutional layer levels. Each level uses a 1×1 convolution kernel to output the multi-channel high-frequency diffusion features and multi-channel low-frequency diffusion features extracted in the current stage. The image corresponding to the high-frequency features is called the detail image, and the image corresponding to the low-frequency features is called the basic image.

[0087] 7) The multi-channel high-frequency diffusion features extracted by the detail adaptive denoising network are fused with high-frequency features through the adaptive regional consistency fusion network. The overall architecture of the adaptive regional consistency fusion network is as follows: Figure 5 As shown. First, a symmetric filling operation is applied to the extracted multi-channel high-frequency diffusion features, a one-pixel border is added, and a 3×3 window is used for maximum filtering. The maximum filter can be expressed as:

[0088] (12)

[0089] in, represents the output image, Indicates The pixel values ​​of the 3×3 matrix centered at Indicates taking the maximum value.

[0090] Then, the sign function is applied to the multi-channel high-frequency diffusion features after maximization to generate an activity map. A 3×3 convolution layer is used to map the high-dimensional fusion features to the 128-channel output. The threshold of each pixel is calculated according to the local area of ​​the output, and the mapping matrix is ​​generated according to the adaptive threshold. The calculation process can be expressed as:

[0091] (13)

[0092] (14)

[0093] in, represents the symbolic function, and represents different filtered images, represents the mapping value of the convolution matrix, Represents the mapping matrix.

[0094] Finally, replace the parameter 0 in the mapping matrix with the pixel corresponding to the infrared image, and replace the parameter 1 with the pixel corresponding to the visible light image, and then map it through the convolution layer to obtain the fused detail image. .

[0095] 8) The multi-channel low-frequency diffusion features extracted by the detail adaptive denoising network are fused with low-frequency features through a multi-channel low-frequency feature fusion network. The architecture of the multi-channel low-frequency feature fusion network is as follows: Figure 6 First, these multi-channel low-frequency features are processed by combining nearest neighbor upsampling and 1×1 convolution kernels, and then a series of convolution layers with 3×3 convolution kernels are used to map the low-dimensional information features to 3-channel outputs to obtain the fusion base map.

[0096] 9) The Concat strategy is used to fuse the fusion detail map dominated by high-frequency features obtained by the adaptive regional consistency fusion network and the fusion basic map dominated by low-frequency features obtained by the multi-channel low-frequency feature fusion network to reconstruct the fusion image of infrared and visible light.

[0097] The key algorithm of the technical solution of the present invention is the conditional diffusion model fusion algorithm. Compared with the currently popular infrared and visible light image fusion algorithm, this algorithm has made significant improvements in color fidelity, texture detail preservation and fusion quality. It not only completes the feature extraction and fusion tasks by combining the conditional denoising diffusion implicit model and the multi-channel high-low frequency parallel fusion model, but also alleviates the problem of the lack of benchmark true values ​​of the fusion results when the denoising diffusion implicit model performs image fusion tasks. It also further demonstrates the unique advantages of the denoising diffusion implicit model in image processing, and still performs well in generating high-quality images and mathematical interpretability, realizing the optimization of existing methods and the upgrading of existing technologies.

[0098] The description should be combined with the technical solution and be well-reasoned (provide necessary experimental data); try to quantify what can be quantified, and for what cannot be quantified, describe it by analyzing the structural characteristics and theoretical explanation.

[0099] Comparison of experimental data with evaluation results

[0100] To prove the effectiveness of this project, the public dataset MSRS was selected as the source image in the experiment to perform the infrared and visible light image fusion task, and the public dataset ReodScene was selected to perform the model generalization verification. Compared with 9 advanced image fusion methods in recent years, the comparison algorithms are: an infrared and visible light image fusion algorithm based on residual network, an image fusion algorithm based on compression decomposition network, a unified unsupervised end-to-end image fusion algorithm based on adaptive similarity, a multimodal image fusion algorithm based on dual-branch feature decomposition of association, an infrared and visible light image fusion algorithm based on dual adversarial learning, an image fusion algorithm based on denoising diffusion probability model, a low-light infrared and visible light image fusion algorithm based on scene illumination disentanglement network, a general image fusion algorithm based on cross-domain remote learning and Swin-Transformer, and an infrared and visible light image fusion algorithm based on guided filter and potential low-rank representation.

[0101] The infrared and visible light image fusion effect evaluation method of the present invention has subjective visual evaluation and objective evaluation indicators. In order to fully reflect that the algorithm framework of this paper has good fusion performance, eight objective evaluation indicators are selected to measure the fusion results, including entropy (EN), mutual information (MI), structural similarity (SSIM), peak signal-to-noise ratio (PSNR), average gradient (AG), spatial frequency (SF), correlation coefficient (CC), and visual fidelity (VIFF). AG and EN are important indicators for measuring image clarity and information content. The larger the value, the richer the information extracted from the input image. MI is used to measure the similarity between the fused image and the source image, that is, how much information the fused image obtains from the source image. The larger the mutual information, the better the fusion quality. CC is used to measure the spatial linear correlation between the fused image and the source image. The closer its value is to 1 or -1, the more information the fused image contains from the source image, and the better the fusion effect. The higher the SSIM value, the better the consistency between the fusion result and the visual characteristics of the human eye. PSNR reflects whether the image is distorted by measuring the ratio between the effective information and the noise of the image. The larger the value, the better the image fusion quality. SF is used to measure the change of image grayscale. The larger the SF value, the clearer the image and the better the fusion quality. VIFF is an estimate of the information extracted from the fused image. The larger the value, the higher the consistency with the subjective characteristics of the human eye and the better the image fusion effect quality. The experimental data of three groups of images with different infrared and visible light sources in the two data sets are shown in Tables 1, 2, and 3.

[0102] Table 1 Objective evaluation indicators of night scenes in the MSRS dataset “00004N”

[0103]

[0104] Table 2 Objective evaluation indicators of daytime scenes in the MSRS dataset “00241D”

[0105]

[0106] Table 3 Objective evaluation indicators of the RoadScene dataset “FLIR_0579”

[0107]

[0108] It can be seen from the eight objective evaluation indicators in Tables 1, 2 and 3 that the objective evaluation indicators of EN, MI, AG, SF, SSIM, PSNR, CC and VIFF of the infrared and visible light image fusion algorithm proposed in the present invention are improved by 4.52%, 46.29%, 17.77%, 29.92%, 24.14%, 16.23%, 8.52% and 59.62% on average compared with the other nine high-level algorithms, indicating that the fusion algorithm is superior to other algorithms in terms of color fidelity, texture detail preservation and structural similarity, and achieves a good balance between feature information extraction and high-frequency noise suppression. It has strong generalization and can handle infrared and visible light image fusion tasks under various lighting and environmental scenes. The AG and SF evaluation indicators of this algorithm are slightly lower than those of algorithms DIVFusion, U2Fusion and BTSFusion, because this algorithm uses a detail adaptive denoising network to achieve a delicate balance between texture detail enhancement and noise reduction effects, and does not overly focus on the brightness information of the image. For the MI index, the proposed algorithm is significantly higher than DIVFusion, U2Fusion and BTSFusion, which indicates that the proposed algorithm has a strong ability to extract image information and achieves a good balance between brightness information and texture detail preservation. The remaining evaluation indicators are higher than the comparison algorithms, which indicates that the image fusion effect of the proposed method is excellent.

[0109] The technical solution of the present invention focuses on the conditional diffusion fusion model. Compared with the currently popular image fusion algorithm, the performance of this algorithm has been significantly improved. The conditional denoising diffusion implicit algorithm further demonstrates the unique advantages of the denoising diffusion implicit model in image processing on the original basis, and still performs well in generating high-quality images and mathematical interpretability. It not only solves the problem that the denoising diffusion model lacks a benchmark true value in the infrared and visible light image fusion task, and it is difficult to directly realize the image fusion task, but also achieves a good balance between feature information extraction and high-frequency noise suppression, thereby improving the fusion quality. The multi-channel high-low frequency parallel fusion algorithm processes the high-frequency features and low-frequency features in the image with appropriate fusion rules through the adaptive regional consistency fusion network and the multi-channel low-frequency feature fusion network, respectively, retaining rich and consistent texture details and scene physical information. The overall fusion model has strong mathematical interpretability, high stability, and good fusion quality, realizing the optimization of existing methods and the upgrading of existing technologies.

[0110] According to the above experimental data and analysis and evaluation, the technical effects of the present invention are specifically manifested in the following aspects:

[0111] ① The conditional diffusion fusion model combines a variety of image fusion-related algorithms to achieve infrared and visible light image fusion. Image fusion consists of nine steps and five algorithms.

[0112] ② A multi-channel likelihood correction network is proposed. The conditional denoising diffusion implicit model is used as the feature extraction network. The stitching technology is used to train the stitched source image as the benchmark truth value. At the same time, a multi-channel likelihood correction module is introduced in the reverse denoising process. Through the stitching technology and the multi-channel likelihood correction module, the highest priority distribution suitable for the feature extraction task of infrared and visible light images is obtained, and the prediction of the generated results is continuously corrected in the iterative process, so as to more accurately simulate the multi-channel complex distribution of infrared and visible light images.

[0113] ③ A detail adaptive denoising network is proposed. Through fully connected skip connections and semantic and detail enhancement modules, the decoding layer is endowed with rich semantic features while retaining delicate texture details. This enables the denoising network to achieve a delicate balance between texture detail enhancement and noise reduction effects, and effectively learns and integrates high-frequency and low-frequency features.

[0114] ④ Adopting the adaptive regional consistency fusion network as the detail image fusion network, the high-frequency information is fused through the consistency verification scheme. This fusion relies on a binary mapping matrix, which is obtained by regional consistency verification using a 3×3 adaptive threshold window, ensuring that adjacent pixels are fused in a similar form and retaining the texture details of the image.

[0115] A multi-channel low-frequency feature fusion network is used as the basic image fusion network. The adaptive region consistency fusion network and the multi-channel low-frequency feature fusion network process the high-frequency features and low-frequency features in the image with appropriate fusion rules respectively, maintaining rich and consistent texture details and scene physical information.

[0116] Figures 7 to 9 The fusion comparison diagrams of different methods for different scenes are shown in Figure 1. The sub-images Infrared and Visible are the original infrared image and visible light image; the sub-image RFN-Nest is a residual fusion algorithm for infrared and visible light images based on residual fusion network (RFN); SDNet is an image fusion algorithm based on squeeze and decomposition network (SD); U2Fusion is a unified unsupervised end-to-end image fusion algorithm based on adaptive similarity; CDDFuse is a correlation-driven dual branch feature decomposition (CDDFuse). CDD) is a multimodal image fusion algorithm; TarDAL is an infrared and visible light image fusion algorithm based on dual adversarial learning; DDFM is an image fusion algorithm based on a denoising diffusion probability model; DIVFusion is a low-light infrared and visible light image fusion algorithm based on a scene illumination disentanglement network; SwinFusion is a general image fusion algorithm based on cross-domain remote learning and Swin-Transformer; BTSFusion is an infrared and visible light image fusion algorithm based on guided filters and potential low-rank representation; Ours is the result graph of the infrared and visible light image fusion algorithm of the present invention. In summary, the fused image obtained by the method of the present invention is significantly better than other fusion algorithms in terms of subjective perception and objective evaluation, which shows that the experimental effect of the method of the present invention is better.

[0117] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0118] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A method for fusion of infrared and visible light images based on a conditional diffusion model, characterized in that: The specific implementation steps are as follows: S1. The registered single-channel infrared image and three-channel visible light image are spliced ​​into a four-channel image and input into the conditional diffusion module. The prior distribution of the custom subset is approximated as the prior distribution suitable for the infrared and visible light image feature extraction task. In the forward diffusion process of the conditional diffusion module, the model gradually adds Gaussian noise in the form of a Markov chain within T steps. The multi-channel noisy image Ft is expressed as: (1) in, represents the standard normal distribution, and Respectively represent the multi-channel noisy image and the original image of the current step length, 1- represents the product of all noise variances before step size t, express Obey Gaussian distribution; In the reverse denoising process of the conditional diffusion module, the model gradually removes noise through T time steps. At time step t, the multi-channel source image The prediction of , which can be expressed as: (2) Furthermore, in the case of known multi-channel noisy images and a multi-channel source image When , the multi-channel noisy image of the previous time step can be derived , which can be expressed as: (3) in, represents the noise variance at time step t, Represents a noisy image The mean of the conditional distribution of can be expressed as: (4) in, represents the denoising network, and They represent the input noisy multi-channel image and time step of the denoising network respectively; S2. Utilize the highest a priori distribution learned by the conditional diffusion model, combine the denoising generation process and the multi-channel likelihood correction network to extract the multi-channel feature distribution of the spliced ​​source image; set the latent space to introduce the multi-channel likelihood correction network in the reverse denoising generation process, and at each time step, Prediction The spliced ​​four-channel image is combined with the multi-channel EM algorithm for likelihood correction. The semantic information is gradually introduced in the iterative process, so that the denoising result can better retain the characteristic information of the source image. The diffusion modeling in the latent space is used to achieve accurate simulation of the multi-channel distribution of infrared and visible light images. The calculation process can be expressed as: (5) in, Represents a multi-channel source image The difference between the prediction and the visible light signature, Represents a multi-channel source image The difference between the prediction and the infrared signature, and represents the latent space scale parameter, represents the penalty term; S3, intercepting the generation results of three different time steps in the denoising generation process of the conditional diffusion model, applying a splicing operation to them to obtain multi-stage splicing features; S4. The multi-stage infrared and visible light features generated by the denoising generation process are used through a detail adaptive denoising network to extract four-channel image features and separate high- and low-frequency features. First, the spliced ​​image of each time step is used as the input image, and a 3×3 convolution kernel is used to perform preliminary feature extraction and channel number alignment on the input image; then, a deep neural network encoder is used to extract hierarchical and abstract features, and the channels are aligned. The hierarchical feature map generated by the encoder is used to apply the spatial and channel attention mechanism to each layer of the feature map. , which can be expressed as: (6) in, and Respectively represent the encoder Feature maps before and after layer processing, and Respectively represent the encoder Parameters of layer channels and spatial attention, Represents a convolution with a kernel size of 3, which is used to enhance the aggregation ability of spatial and channel attention; Finally, the feature map extracted by the encoder is fed into the decoder, and the semantic and detail enhancement modules are used to connect the encoder and decoder stages at each level; , select the corresponding feature map For reference, the size of feature maps at different levels is adjusted using up and down sampling to fit the encoder feature map The resolution is calculated and element-by-element multiplication is performed. The center frequency domain filtering is performed in the frequency domain using Fourier transform to selectively filter out the low-frequency components in the feature map and retain the multi-channel high-frequency diffusion feature output. The calculation process can be expressed as: (7) (8) (9) in, , and Represent the adaptive average pooling, unit mapping and nearest neighbor upsampling operations respectively, and Decoder Feature maps before and after layer processing, and Respectively The height and width of represents element-wise multiplication, applied to all feature maps of adjusted resolution, and denote Fourier transform and inverse Fourier transform respectively, is the Fourier mask, is the center filter radius, is the threshold frequency; S5, through the current level feature map in the denoising network decoder stage Calculate the average value of the backbone feature map to achieve balanced adjustment of the extracted features; To avoid the extracted high-frequency features from producing too smooth textures, use the full-channel scaling operation only in the first layer of the current level, and limit the subsequent level scaling operations to The half channel retains the multi-channel low-frequency diffusion characteristic output. The calculation process can be expressed as: (10) (11) in, and Represents feature maps The number of channels and the current decoder level, express No. channels; S6. Output the image diffusion features extracted by the detail adaptive denoising network. The decoder of the detail adaptive denoising network includes 5 convolution layer levels. Each level uses a 1×1 convolution kernel to output the multi-channel high-frequency diffusion features and multi-channel low-frequency diffusion features extracted at the current stage. The map corresponding to the high-frequency features is called the detail map, and the map corresponding to the low-frequency features is called the basic map. S7. The multi-channel high-frequency diffusion features extracted by the detail adaptive denoising network are fused through the adaptive regional consistency fusion network. First, a symmetric filling operation is applied to the extracted multi-channel high-frequency diffusion features, a one-pixel border is added, and a 3×3 window is used for maximum filtering. The maximum filter can be expressed as: (12) in, represents the output image, Indicates The pixel values ​​of the 3×3 matrix centered at Indicates taking the maximum value; Then, the sign function is applied to the multi-channel high-frequency diffusion features after maximization to generate an activity map. A 3×3 convolution layer is used to map the high-dimensional fusion features to the 128-channel output. The threshold of each pixel is calculated according to the local area of ​​the output, and the mapping matrix is ​​generated according to the adaptive threshold. The calculation process can be expressed as: (13) (14) in, represents the symbolic function, and represents different filtered images, represents the mapping value of the convolution matrix, represents the mapping matrix; Finally, replace the parameter 0 in the mapping matrix with the pixel corresponding to the infrared image, and replace the parameter 1 with the pixel corresponding to the visible light image, and then map it through the convolution layer to obtain the fused detail image. ; S8. The multi-channel low-frequency diffusion features extracted by the detail adaptive denoising network are subjected to low-frequency feature fusion by the multi-channel low-frequency feature fusion network. These multi-channel low-frequency features are first processed by combining the nearest neighbor upsampling and 1×1 convolution kernel, and then a series of convolution layers with a convolution kernel of 3×3 are used to map the low-dimensional information features to 3-channel outputs to obtain a fusion base map. S9. The Concat strategy is used to fuse the fusion detail map dominated by high-frequency features obtained by the adaptive regional consistency fusion network and the fusion basic map dominated by low-frequency features obtained by the multi-channel low-frequency feature fusion network to reconstruct the fusion image of infrared and visible light.

Images