An infrared and visible light image fusion method based on a conditional diffusion model

Abstract

The application discloses an infrared and visible light image fusion method based on a conditional diffusion model and belongs to the technical field of image fusion. Compared with the current popular infrared and visible light image fusion algorithm, the algorithm has obvious improvement in color fidelity, texture detail preservation and fusion quality. The algorithm not only combines a conditional denoising diffusion implicit model and a multi-channel high-low frequency parallel fusion model to complete feature extraction and fusion tasks, alleviates the problem that the denoising diffusion implicit model lacks a benchmark true value of a fusion result when performing an image fusion task, but also further exhibits the unique advantages of the denoising diffusion implicit model in image processing, performs well in generating high-quality images and mathematical interpretability, and realizes optimization of existing methods and upgrading of existing technologies.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image fusion, and particularly relates to an infrared and visible light image fusion method based on a conditional diffusion model. BACKGROUND

[0002] Infrared and visible light image fusion can well fuse features from multiple types of sensors to generate high-quality fusion images with comprehensive scene information. Infrared images can capture thermal radiation information and have significant contrast, and visible light images contain rich detail information and have high resolution, but single-mode image information is single and has its own shortcomings. In order to make the fusion result more consistent with human visual perception and provide more accurate semantic representation, infrared and visible light image fusion is widely used in remote sensing, all-weather detection, military reconnaissance and other fields. The fused image combines the thermal radiation information in the infrared image and the texture details of the visible light image, overcoming the limitations of visible light images being easily affected by the environment and infrared images lacking texture information.

[0003] The purpose of infrared and visible light image fusion is to use a trained network model to extract features without manually formulating artificial levels and measurement rules. By selecting an appropriate fusion strategy, the advantages of different imaging principle information are played, the shortcomings of one imaging principle information are made up, and the accuracy, comprehensiveness and usability of the obtained information are improved, providing more effective and comprehensive information for all-weather detection and other tasks.

[0004] The core of the improvement of infrared and visible light image fusion technology lies in optimizing the algorithm and improving the computing power. At present, due to its powerful feature extraction capability, deep learning has become the mainstream algorithm in the current image fusion field. Deep learning algorithms mostly constrain the fusion image by constructing an objective function, and rely on the powerful nonlinear fitting capability of neural networks to obtain the expected distribution characteristics. Among them, the image fusion framework based on GAN network has produced good fusion results, but there are still problems such as unstable training process and lack of interpretability. In recent years, due to the stability and controllability of the picture generation process, the research on denoising diffusion probability model has received widespread attention.

[0005] Deep learning algorithms based on diffusion models have great advantages in image fusion, and perform well in generating high-quality images and mathematical interpretability. However, there are still major challenges, and it is difficult to directly implement image fusion tasks due to the lack of benchmark true values of fusion results. To solve this problem, some methods use pre-trained diffusion models to provide prior knowledge, and realize image fusion by manually constructing true values. However, the pre-trained diffusion weights used are usually not suitable for the current image fusion task, resulting in insufficient feature extraction, loss of detail information in the fused image, poor quality, and the structure of the pre-trained network limits the flexibility and application range of the model. In order to improve the fusion quality, a conditional denoising diffusion model can be used to solve the problem of lack of true value fusion image.

[0006] The application provides an infrared and visible light image fusion method based on a conditional diffusion model, which can significantly enhance the texture details and edge contour information of the fused image, and has excellent visual effect and color fidelity, and improves the application in the fields of military, all-weather monitoring, medical technology and the like. SUMMARY

[0007] The application provides an infrared and visible light image fusion method based on a conditional diffusion model.

[0008] To achieve the above object, the technical scheme adopted by the application is as follows:

[0009] 1) The single-channel infrared image and the three-channel visible light image that have been registered are spliced into a four-channel image input into a conditional diffusion module, the prior distribution of a self-defined subset is approximated to a prior distribution suitable for the feature extraction task of infrared and visible light images, in the forward diffusion process of the conditional diffusion module, the model gradually adds Gaussian noise in T steps in the form of Markov chain, and the multi-channel noisy image Ft is represented as:

[0010] (1)

[0011] wherein, represents a standard normal distribution, and respectively represent the multi-channel noisy image and the original image of the current step, 1- represents the product of the variances of all noises before step t, represents obeys a Gaussian distribution.

[0012] In the reverse denoising process of the conditional diffusion module, the model gradually removes noise through T time steps, and at time step t, the prediction of the multi-channel source image is recorded as , and can be represented as:

[0013] (2)

[0014] Further, given a multi-channel noisy image and a multi-channel source image , the multi-channel noisy image at the previous time step can be denoted as:

[0015] (3)

[0016] where denotes the noise variance at time step t, denotes the mean of the conditional distribution of the noisy image , which can be denoted as:

[0017] (4)

[0018] where denotes the denoising network, and denote the input noisy multi-channel image and the time step of the denoising network, respectively;

[0019] 2) Using the optimal prior distribution learned by the conditional diffusion model, combine the denoising generation process and the multi-channel likelihood correction network to extract the multi-channel feature distribution of the spliced source image; In the reverse denoising generation process, introduce the multi-channel likelihood correction network in the latent space, and in each time step, the prediction of the multi-channel source image is corrected using the spliced four-channel image combined with the multi-channel EM algorithm. Gradually introduce semantic information in the iteration process, so that the denoising generation result can better preserve the feature information of the source image. Through diffusion modeling in the latent space, accurate simulation of the multi-channel distribution of infrared and visible light images is realized. The calculation process can be denoted as:

[0020] (5)

[0021] where denotes the difference between the prediction of the multi-channel source image and the visible light feature, denotes the difference between the prediction of the multi-channel source image and the infrared feature, and denote the latent space scale parameters, denote the penalty term;

[0022] 3) The generation results of three different time steps in the denoising generation process of the truncated condition diffusion model are obtained by using splicing operation, and the spliced features of multiple stages are obtained;

[0023] 4) The multi-stage infrared and visible light features generated by the denoising generation process are subjected to four-channel image feature extraction and high and low frequency feature separation through the detail adaptive denoising network. First, the spliced images of each time step are used as input images, and a 3x3 size convolution kernel is used for preliminary feature extraction and channel number alignment of the input images. Then, the deep neural network encoder is used to extract hierarchical and abstract features and perform channel number alignment. The hierarchical feature maps generated by the encoder are applied to each layer feature map , which can be expressed as:

[0024] (6)

[0025] wherein, and represent the feature maps before and after the processing of the layer of the encoder, and represent the parameters of the channel and spatial attention of the layer of the encoder, represents a convolution with a kernel size of 3, which is used to enhance the aggregation ability of spatial and channel attention.

[0026] Finally, the feature maps extracted by the encoder are sent to the decoder, and the semantic and detail enhancement module is used to connect each level of the encoder and the decoder stage. For different levels of the encoder , the corresponding feature map is selected as a reference, and the size of the feature map of different levels is adjusted by using up-sampling to adapt to the resolution of the encoder feature map , and element-wise multiplication is performed, Fourier transform is used to perform center frequency filtering in the frequency domain, and low frequency components in the feature map are selectively filtered out, and multi-channel high frequency diffusion features are output. The calculation process can be expressed as:

[0027] (7)

[0028] (8)

[0029] (9)

[0030] wherein, , and represent adaptive average pooling, unit mapping and nearest neighbor up-sampling operations, and respectively represent the decoder feature maps before and after layer processing, and respectively represent height and width, represent element-wise multiplication, applied to all resolution-adjusted feature maps, and respectively represent Fourier transform and inverse Fourier transform, is a Fourier mask, is a center filter radius, is a threshold frequency;

[0031] 5) In the denoising network decoder stage, the average value of the current level feature map is calculated to balance the extracted features; to avoid the extracted high-frequency features producing overly smooth textures, only the first layer of the current level uses full-channel scaling operations, and subsequent layer scaling operations are limited to half channels of , preserving multi-channel low-frequency diffusion feature output, and the calculation process can be represented as:

[0032] (10)

[0033] (11)

[0034] wherein, and respectively represent the number of channels of the feature map and the current decoder level to which it belongs, represents the channel of ;

[0035] 6) The image diffusion features extracted by the detail adaptive denoising network are output, and the decoder of the detail adaptive denoising network includes 5 convolution levels, each level uses a 1×1 convolution kernel to output the multi-channel high-frequency diffusion features and the multi-channel low-frequency diffusion features extracted in the current stage, and the image corresponding to the high-frequency features is called a detail image, while the image corresponding to the low-frequency features is called a base image;

[0036] 7) The multi-channel high-frequency diffusion features extracted by the detail adaptive denoising network are fused by the adaptive region consistency fusion network, first, the multi-channel high-frequency diffusion features extracted are applied to the symmetric padding operation to increase the border by one pixel, and a 3×3 window is used for maximum filtering, and the maximum filter can be represented as:

[0037] (12)

[0038] wherein, represents an output image, represents the pixel value of a 3x3 matrix centered at represents taking the maximum value.

[0039] Then, the multi-channel high-frequency diffusion feature after the maximum processing is applied to the sign function to generate an activity map, a 3x3 convolution layer is used to map the high-dimensional fusion feature to 128-channel output, the threshold value of each pixel is calculated according to the local area of the output, and a mapping matrix is generated according to the adaptive threshold value, and the calculation process can be represented as:

[0040] (13)

[0041] (14)

[0042] wherein, represents a sign function, and represents different filter images, represents the mapping value of the convolution matrix, represents a mapping matrix.

[0043] Finally, the parameter 0 in the mapping matrix is replaced by the pixel corresponding to the infrared image, and the parameter 1 is replaced by the pixel corresponding to the visible light image, and then the fused detail map is obtained through the convolution layer mapping .

[0044] 8) The multi-channel low-frequency diffusion feature extracted by the detail adaptive denoising network is subjected to low-frequency feature fusion through a multi-channel low-frequency feature fusion network, and the multi-channel low-frequency feature is subjected to combination processing by adopting nearest neighbor up-sampling and 1x1 convolution kernel, and then a series of convolution layers with a 3x3 convolution kernel are used to map the low-dimensional information feature to 3-channel output, so as to obtain a fused basic map.

[0045] 9) The fused detail map dominated by high-frequency features obtained through the adaptive region consistency fusion network and the fused basic map dominated by low-frequency features obtained through the multi-channel low-frequency feature fusion network are fused by using a Concat strategy to reconstruct an infrared and visible light fusion image.

[0046] Compared with the prior art, the present application has the following beneficial effects:

[0047] ​The key algorithm of the technical scheme of the present application is a conditional diffusion model fusion algorithm; compared with the current popular infrared and visible light image fusion algorithm, the algorithm has obvious improvement in color fidelity, texture detail preservation and fusion quality, not only completes the feature extraction and fusion task by combining the conditional denoising diffusion implicit model and the multi-channel high-low frequency parallel fusion model, alleviates the problem that the denoising diffusion implicit model lacks a benchmark true value of the fusion result when performing the image fusion task, but also further exhibits the unique advantages of the denoising diffusion implicit model in image processing, still performs well in generating high-quality images and mathematical interpretability, and realizes the optimization of the existing method and the upgrading of the existing technology. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 The infrared and visible light image fusion method flowchart of the conditional diffusion model of the present application is shown in the figure;

[0049] Figure 2 The multi-channel likelihood correction module of the present application is shown in the figure;

[0050] Figure 3 The detail adaptive denoising network of the present application is shown in the figure;

[0051] Figure 4 The semantic and detail enhancement module of the present application is shown in the figure;

[0052] Figure 5 The adaptive region consistency fusion network of the present application is shown in the figure;

[0053] Figure 6 The multi-channel low-frequency feature fusion network of the present application is shown in the figure;

[0054] Figure 7 The MSRS dataset "00004N" night scene fusion result of the present application is shown in the figure;

[0055] Figure 8 The MSRS dataset "00241D" daytime scene fusion result of the present application is shown in the figure;

[0056] Figure 9 The RoadScene dataset "FLIR_0579" scene fusion result of the present application is shown in the figure. DETAILED DESCRIPTION

[0057] In order to make the technical means, creative features, purposes and effects realized by the present application easy to understand, the present application is further described below in combination with specific embodiments.

[0058] The infrared and visible light image fusion method flowchart of the technical scheme of the present application is shown in the figure as follows: Figure 1

[0059] The specific implementation steps are as follows: ​

[0060] 1) The single-channel infrared image and three-channel visible light image that have been registered are spliced into a four-channel image input condition diffusion module, the prior distribution of the custom subset is approximated as a prior distribution suitable for the feature extraction task of infrared and visible light images, so that the denoising generation process can be more targeted according to the characteristics of the task, so that the diffusion process is more compact and more suitable for the feature extraction task of infrared and visible light image fusion. In the forward diffusion process of the conditional diffusion module, the model gradually adds Gaussian noise in T steps in the form of a Markov chain, and the multi-channel noisy image Ft is represented as:

[0061] (1)

[0062] wherein, represents a standard normal distribution, and represent the multi-channel noisy image and the original image at the current step, respectively, and 1- represents the product of all noise variances before step t, represents obeys a Gaussian distribution.

[0063] In the reverse denoising process of the conditional diffusion module, the model gradually removes noise through T time steps, and at time step t, the prediction of the multi-channel source image is denoted as , which can be represented as:

[0064] (2)

[0065] Further, when the multi-channel noisy image and the multi-channel source image are known, the multi-channel noisy image at the previous time step can be derived and represented as:

[0066] (3)

[0067] wherein, represents the noise variance at time step t, represents the mean of the conditional distribution of the noisy image , which can be represented as:

[0068] (4)

[0069] wherein, represents a denoising network, and represent the input noisy multi-channel image and the time step of the denoising network, respectively.

[0070] 2) Using the optimal prior distribution learned by the conditional diffusion model, the multi-channel feature distribution of the spliced source image is extracted by combining the denoising generation process and the multi-channel likelihood correction network. In the reverse denoising generation process, the latent space is introduced to the multi-channel likelihood correction network, and the network architecture is as shown in Figure 2 . At each time step, the prediction of the multi-channel source image is spliced with the four-channel image, and the multi-channel EM algorithm is used for likelihood correction. In the iteration process, semantic information is gradually introduced, so that the denoising generation result can better retain the feature information of the source image. Through diffusion modeling in the latent space, the multi-channel distribution of infrared and visible light images is accurately simulated, and the calculation process can be represented as:

[0071] (5)

[0072] Wherein, represents the difference between the prediction of the multi-channel source image and the visible light feature, represents the difference between the prediction of the multi-channel source image and the infrared feature, and represent the latent space scale parameters, represents the penalty term.

[0073] 3) The generation results of three different time steps in the denoising generation process of the conditional diffusion model are intercepted, and the multi-stage spliced features are obtained by splicing operation.

[0074] 4) The multi-stage infrared and visible light features generated by the denoising generation process are extracted and the high and low frequency features are separated through the detail adaptive denoising network. The overall architecture of the detail adaptive denoising network is as shown in Figure 3 . First, the spliced image of each time step is taken as the input image, and a 3x3 size convolution kernel is used to perform preliminary feature extraction and channel number alignment on the input image. Then, the deep neural network encoder is used to extract hierarchical and abstract features and perform channel number alignment. The hierarchical feature maps generated by the encoder are applied to each layer of feature map , which can be represented as:

[0075] (6)

[0076] Wherein, and represent the feature maps before and after the layer of the encoder, and represent the feature maps before and after the Parameters of layer channel and spatial attention, represents a convolution with kernel size 3, which is used to enhance the aggregation ability of spatial and channel attention.

[0077] Finally, the encoder extracted feature maps are sent to the decoder, and the semantic and detail enhancement module is used to connect the encoder and decoder stages of each level. The architecture of the semantic and detail enhancement module is shown in Figure 4 For different levels of the encoder , the corresponding feature maps are selected as references, and the size of the feature maps of different levels is adjusted by upsampling and downsampling to adapt to the resolution of the encoder feature maps , and element-wise multiplication is performed, and center frequency domain filtering is performed in the frequency domain by Fourier transform, and the low frequency components in the feature maps are selectively filtered out, and multi-channel high frequency diffusion features are output, and the calculation process can be represented as:

[0078] (7)

[0079] (8)

[0080] (9)

[0081] where, , and represent adaptive average pooling, unit mapping and nearest neighbor upsampling operations respectively, and represent the feature maps before and after the processing of the layer of the decoder, and represent the height and width of , represents element-wise multiplication, which is applied to all feature maps with adjusted resolution, and represent Fourier transform and inverse Fourier transform respectively, is the Fourier mask, is the center filter radius, is the threshold frequency.

[0082] 5) In the decoder stage of the denoising network, the average value of the backbone feature map is calculated based on the current level feature map to achieve balanced adjustment of the extracted features. To avoid the extracted high frequency features producing overly smooth textures, only the full channel scaling operation is used in the first layer of the current level, and the scaling operation of the subsequent layers is limited to half channel, and multi-channel low frequency diffusion features are output, and the calculation process can be represented as:

[0083] (10)

[0084] (11)

[0085] wherein, and respectively represent the number of channels of feature map and the current decoder level to which it belongs, represents the channel of .

[0086] 6) The image diffusion features extracted by the detail adaptive denoising network are output, and the decoder of the detail adaptive denoising network includes 5 convolution levels. Each level uses a 1x1 convolution kernel to output the multi-channel high-frequency diffusion features and multi-channel low-frequency diffusion features extracted in the current stage, respectively. The image corresponding to the high-frequency features is called a detail image, and the image corresponding to the low-frequency features is called a base image.

[0087] 7) The multi-channel high-frequency diffusion features extracted by the detail adaptive denoising network are fused by the adaptive region consistency fusion network. The overall architecture of the adaptive region consistency fusion network is shown in Figure 5 . First, the multi-channel high-frequency diffusion features extracted are applied to the symmetric padding operation to increase the border of one pixel, and a 3x3 window is used for maximum filtering. The maximum filter can be represented as:

[0088] (12)

[0089] wherein, represents the output image, represents the pixel value of the 3x3 matrix centered on , and represents taking the maximum value.

[0090] Then, the multi-channel high-frequency diffusion features after the maximum processing are applied to the sign function to generate an activity map. A 3x3 convolution layer is used to map the high-dimensional fusion features to an output of 128 channels. The threshold value of each pixel is calculated according to the local region of the output, and a mapping matrix is generated according to the adaptive threshold value. The calculation process can be represented as:

[0091] (13)

[0092] (14)

[0093] wherein, represents the sign function, and represent different filtered images, represents the mapping value of the convolution matrix, The mapping matrix is represented.

[0094] Finally, the parameter 0 in the mapping matrix is replaced by the pixel corresponding to the infrared image, and the parameter 1 is replaced by the pixel corresponding to the visible light image, and then the convolution layer mapping is performed to obtain the fusion detail map .

[0095] 8) The multi-channel low-frequency diffusion features extracted by the detail adaptive denoising network are subjected to low-frequency feature fusion through a multi-channel low-frequency feature fusion network, and the architecture of the multi-channel low-frequency feature fusion network is as shown in Figure 6 . First, these multi-channel low-frequency features are combined and processed by nearest neighbor up-sampling and 1x1 convolution kernel, and then a series of convolution layers with 3x3 convolution kernel are used to map the low-dimensional information features to 3-channel output to obtain the fusion base map.

[0096] 9) The fusion detail map dominated by high-frequency features obtained by the adaptive region consistency fusion network and the fusion base map dominated by low-frequency features obtained by the multi-channel low-frequency feature fusion network are fused by using the Concat strategy to reconstruct the infrared and visible light fusion image.

[0097] The key algorithm of the technical solution of the present application is the conditional diffusion model fusion algorithm. Compared with the current popular infrared and visible light image fusion algorithm, the algorithm has obvious improvement in color fidelity, texture detail preservation and fusion quality. Not only does it complete the feature extraction and fusion task by combining the conditional denoising diffusion implicit model and the multi-channel high-low frequency parallel fusion model, but also it alleviates the problem of lack of benchmark true value of the fusion result when the denoising diffusion implicit model is executed in the image fusion task, and further exhibits the unique advantages of the denoising diffusion implicit model in image processing, and still performs well in generating high-quality images and mathematical interpretability, realizing the optimization of existing methods and upgrading of existing technologies.

[0098] In combination with the technical solution, it is ensured that there is reason and evidence (necessary experimental data are provided); as much as possible, the quantifiable is quantified, and the unquantifiable can be described by analyzing the structural characteristics and theoretical explanation.

[0099] Comparison of experimental data and evaluation results

[0100] To prove the effectiveness of the project, the public data set MSRS is selected as the source image in the experiment to perform the infrared and visible light image fusion task, and the public data set ReodScene is selected to verify the model generalization. Compared with nine kinds of image fusion methods more advanced in recent years, the comparison algorithms are: an infrared and visible light image fusion algorithm based on residual network, an image fusion algorithm based on compression decomposition network, a unified unsupervised end-to-end image fusion algorithm based on adaptive similarity, a multi-modal image fusion algorithm based on double-branch feature decomposition of association, an infrared and visible light image fusion algorithm based on double-adversarial learning, an image fusion algorithm based on denoising diffusion probability model, a low-light infrared and visible light image fusion algorithm based on scene lighting disentanglement network, a general image fusion algorithm based on cross-domain remote learning and Swin-Transformer, and an infrared and visible light image fusion algorithm based on guided filter and latent low-rank representation.

[0101] The infrared and visible light image fusion effect evaluation method has subjective visual evaluation and objective evaluation indexes. In order to fully reflect that the algorithm framework in the present application has good fusion performance, eight kinds of objective evaluation indexes are selected to measure the fusion results, including entropy (EN), mutual information (MI), structure similarity (SSIM), peak signal-to-noise ratio (PSNR), average gradient (AG), spatial frequency (SF), correlation coefficient (CC) and visual fidelity (VIFF). AG and EN are important indexes for measuring image definition and information amount, and the larger the value is, the richer the information extracted from the input image is. MI is used to measure the similarity between the fusion image and the source image, that is, how much information the fusion image obtains from the source image, and the larger the mutual information is, the better the fusion quality is. CC is used to measure the spatial linear correlation degree of the fusion image and the source image, and the closer the value is to 1 or -1, the more information the fusion image contains, and the better the fusion effect is. The higher the SSIM value is, the better the consistency of the fusion result and the human eye visual characteristics is. PSNR reflects whether the image is distorted by measuring the ratio between the effective information of the image and the noise, and the larger the value is, the better the image fusion quality is. SF is used to measure the change of image gray scale, and the larger the SF value is, the clearer the image is and the better the fusion quality is. VIFF is an estimate of the information extracted from the fusion image, and the larger the value is, the higher the consistency with the subjective characteristics of the human eye is, and the better the image fusion effect quality is. The experimental data of two data sets, three groups of different infrared and visible light source images, are shown in Tables 1, 2 and 3.

[0102] Table 1 MSRS data set "00004N" night scene objective evaluation index

[0103]

[0104] Table 2 MSRS dataset "00241D" objective evaluation index of daytime scene

[0105]

[0106] Table 3 RoadScene dataset "FLIR_0579" objective evaluation index of scene

[0107]

[0108] As can be seen from the eight objective evaluation indexes in Table 1, Table 2 and Table 3, the EN, MI, AG, SF, SSIM, PSNR, CC and VIFF objective evaluation indexes of the infrared and visible light image fusion algorithm of the present application are improved by 4.52%, 46.29%, 17.77%, 29.92%, 24.14%, 16.23%, 8.52% and 59.62% on average compared with the other nine high-level algorithms, which shows that the fusion algorithm is superior to other algorithms in color fidelity, texture detail preservation and structural similarity, and achieves a good balance between feature information extraction and high-frequency noise suppression, has strong generalization, and can process infrared and visible light image fusion tasks under various light and environmental scenes. The AG and SF evaluation indexes of the algorithm are slightly lower than the algorithms DIVFusion, U2Fusion and BTSFusion, because the algorithm uses a detail adaptive denoising network to achieve a subtle balance between texture detail enhancement and noise reduction effect, and does not pay excessive attention to the brightness information of the image. For the MI index, the algorithm is significantly higher than the algorithms DIVFusion, U2Fusion and BTSFusion, which shows that the algorithm has strong image information extraction capability and achieves a good balance between brightness information and texture detail preservation. The remaining evaluation indexes are higher than the comparison algorithms, which shows that the image effect after image fusion by the method of the present application is excellent.

[0109] Compared with the popular image fusion algorithm, the performance of the algorithm has obviously improved, the conditional denoising diffusion implicit algorithm further shows the unique advantages of the denoising diffusion implicit model in image processing on the original basis, and still performs well in generating high-quality images and mathematical interpretability, not only solves the problem that the denoising diffusion model lacks benchmark true value in the infrared and visible light image fusion task, and directly realizes the image fusion task, but also realizes a good balance between feature information extraction and high-frequency noise suppression, and improves the fusion quality. The multi-channel high-low frequency parallel fusion algorithm processes the high-frequency features and low-frequency features in the image by the adaptive regional consistency fusion network and the multi-channel low-frequency feature fusion network respectively with appropriate fusion rules, and retains rich and consistent texture details and scene physical information. The overall fusion model has strong mathematical interpretability, high stability and good fusion quality, realizes the optimization of the existing method and the upgrading of the existing technology.

[0110] According to the experimental data and analysis and evaluation, the technical effects of the application are specifically shown in the following aspects:

[0111] ① The conditional diffusion fusion model combines a variety of image fusion related algorithms, realizes infrared and visible light image fusion, and the image fusion includes nine steps and five algorithms.

[0112] ② A multi-channel likelihood correction network is proposed, the conditional denoising diffusion implicit model is used as a feature extraction network, the splicing source image is used as a benchmark true value for training by using splicing technology, and a multi-channel likelihood correction module is introduced in the reverse denoising process, through the splicing technology and the multi-channel likelihood correction module, the optimal prior distribution suitable for the infrared and visible light image feature extraction task is obtained, and the prediction of the generated result is continuously corrected in the iteration process, so that the multi-channel complex distribution of the infrared and visible light images is more accurately simulated.

[0113] ③ A detail adaptive denoising network is proposed, through the full-connection skip connection and the semantic and detail enhancement module, rich semantic features are given to the decoding layer while the delicate texture details are retained, so that the denoising network realizes a subtle balance between texture detail enhancement and denoising effect, and effectively learns and integrates high-frequency and low-frequency features.

[0114] ④ An adaptive regional consistency fusion network is used as a detail image fusion network, and high-frequency information is fused through a consistency verification scheme. This fusion depends on a binary mapping matrix, which is obtained by using a 3x3 adaptive threshold window for regional consistency verification, which ensures that adjacent pixels are fused in a similar form and retains the texture details of the image.

[0115] The multi-channel low-frequency feature fusion network is adopted as a basic graph fusion network, and the self-adaptive region consistency fusion network and the multi-channel low-frequency feature fusion network respectively process high-frequency features and low-frequency features in the image with appropriate fusion rules, so that rich and consistent texture details and scene physical information are maintained.

[0116] Figures 7 to 9 Different scene and different method fusion comparison chart, wherein the subgraph Infrared and Visible is the original infrared image and visible light image; the subgraph RFN-Nest is a residual fusion network (RFN) based infrared and visible light image residual fusion algorithm; the SDNet is a squeeze and decomposition network (SD) based image fusion algorithm; the U2Fusion is a unified unsupervised end-to-end image fusion algorithm based on adaptive similarity; the CDDFuse is a multi-modal image fusion algorithm based on correlation driven dual branch feature decomposition (CDD); the TarDAL is an infrared and visible light image fusion algorithm based on double adversarial learning; the DDFM is an image fusion algorithm based on denoising diffusion probability model; the DIVFusion is a low-light infrared and visible light image fusion algorithm based on scene illumination disentanglement network; the SwinFusion is a general image fusion algorithm based on cross-domain remote learning and Swin-Transformer; the BTSFusion is an infrared and visible light image fusion algorithm based on guided filter and latent low-rank representation; and the Ours is the result image of the infrared and visible light image fusion algorithm of the application. In summary, the fusion image obtained by the method of the application is obviously better than that of other fusion algorithms in terms of subjective feeling and objective evaluation, which shows that the experimental effect of the method of the application is better.

[0117] It will be obvious to a person skilled in the art that the application is not limited to the details of the above-described exemplary embodiments, but that the application can be implemented in other concrete forms without departing from the spirit or essential characteristics of the application. The embodiments should therefore be considered in all respects as illustrative and not restrictive, the scope of the application being defined by the appended claims rather than that of the above description, and it is therefore intended that all changes and modifications that fall within the meaning and range of equivalents of the claims be embraced therein. Any reference signs in the claims should not be construed as limiting the claims concerned to the figures in which the reference signs are used.

[0118] Furthermore, it should be understood that although the specification is described in terms of embodiments, not every embodiment includes every feature described. The specification can include implicit combinations of explicitly mentioned features and / or explicit combinations of implicitely mentioned features. Each embodiment depends on the explicit combinations of features and / or the implicit combinations of features made specifically within that embodiment, and each such embodiment can be combined with every other such embodiment to create further embodiments.

Claims

1. An infrared and visible image fusion method based on a conditional diffusion model, characterized in that: The specific implementation steps are as follows: S1, the single-channel infrared image and the three-channel visible light image which have been registered are spliced into a four-channel image input condition diffusion module, the prior distribution of the custom subset is approximated as a prior distribution suitable for the infrared and visible light image feature extraction task, in the forward diffusion process of the condition diffusion module, the model gradually adds Gaussian noise in T steps in the form of Markov chain, and the multi-channel noisy image Ft is represented as: (1) wherein, denotes the standard normal distribution, and denote the multi-channel noisy image and the original image of the current step, respectively, 1- denotes the product of all noise variances up to step t, denotes obeys a Gaussian distribution; In the reverse denoising process of the conditional diffusion module, the model gradually removes noise through T time steps. At time step t, the prediction of the multi-channel source image , denoted as , can be expressed as: (2) Further, when a multi-channel noisy image and a multi-channel source image at a previous time step is known, a multi-channel noisy image at the previous time step can be derived and expressed as: (3) where denotes the noise variance at time step t, denotes the mean of the conditional distribution of the noisy image at time step t. (4) wherein, denotes a denoising network, and denote an input noisy multi-channel image and a time step of the denoising network, respectively; S2, using the optimal prior distribution learned by the conditional diffusion model, combining the denoising generation process and the multi-channel likelihood correction network to splice the source image multi-channel feature distribution extraction; in the reverse denoising generation process, the latent space is introduced into the multi-channel likelihood correction network, and in each time step, the multi-channel source image Prediction The spliced four-channel image is combined with the multi-channel EM algorithm for likelihood correction, and semantic information is gradually introduced in the iteration process, so that the denoising generation result can better retain the feature information of the source image. Through diffusion modeling in the latent space, accurate simulation of the multi-channel distribution of infrared and visible light images is realized, and the calculation process can be represented as: (5) wherein represents a difference between a prediction and a visible feature of the multi-channel source image , represents a difference between a prediction and an infrared feature of the multi-channel source image , and represents a latent space scale parameter, represents a penalty term; S3, the generation results of three different time steps in the denoising generation process of the condition diffusion model are intercepted, and a splicing operation is performed thereon to obtain multi-stage spliced features; S4, the multi-stage infrared and visible light features generated by the denoising generation process are subjected to four-channel image feature extraction and high and low frequency feature separation through a detail adaptive denoising network, first, the spliced images at each time step are taken as input images, and a 3*3 size convolution kernel is used to perform preliminary feature extraction and channel number alignment on the input images; then, a deep neural network encoder is used to extract layered and abstract features and perform channel number alignment, and a spatial and channel attention mechanism is applied to each layer of feature maps generated by the encoder may be represented as: (6) wherein, and respectively represent the first feature maps before and after the layer processing, and respectively represent the first layer channel and spatial attention parameters, represent a convolution with a kernel size of 3, used to enhance the aggregation ability of spatial and channel attention; Finally, the feature maps extracted by the encoder are fed into the decoder, and the semantic and detail enhancement module is used to connect the encoder and decoder stages at each level; for different levels of the encoder , the corresponding feature maps are selected as a reference, the size of the feature map at different levels is adjusted by upsampling and downsampling to adapt to the resolution of the encoder feature map , and element-by-element multiplication is performed, center frequency domain filtering is performed in the frequency domain by Fourier transform, low frequency components in the feature map are selectively filtered out, and multi-channel high frequency diffusion features are reserved, and the calculation process can be represented as: (7) (8) (9) wherein, , and denote adaptive average pooling, unit mapping and nearest-neighbor up-sampling operations, respectively, and denote the feature maps before and after the processing of the decoder’s layer, and denote the height and width of , denotes an element-wise multiplication applied to all resolution-adjusted feature maps, and denote Fourier transform and inverse Fourier transform, respectively, is a Fourier mask, is a center filter radius, is a threshold frequency; S5, passing the current level feature map through the denoising network decoder stage The average value of the main stem feature map is calculated to balance the extracted features. To avoid the extracted high-frequency features producing overly smooth textures, a full-channel scaling operation is used only in the first layer of the current level, and the scaling operation of subsequent layers is limited to half channels, and the multi-channel low-frequency diffusion feature output is reserved. The calculation process can be represented as: (10) (11) wherein, and respectively represent the number of channels of the feature map and the current decoder level to which it belongs, represents the channel of S6, the image diffusion features extracted by the detail adaptive denoising network are output, and the decoder of the detail adaptive denoising network includes 5 convolution levels, each level uses a 1×1 convolution kernel to output the multi-channel high-frequency diffusion features and the multi-channel low-frequency diffusion features extracted in the current stage respectively, the image corresponding to the high-frequency features is called a detail image, and the image corresponding to the low-frequency features is called a base image; S7, the multi-channel high-frequency diffusion features extracted by the detail adaptive denoising network are subjected to high-frequency feature fusion through the adaptive region consistency fusion network, first, the multi-channel high-frequency diffusion features extracted are subjected to a symmetric padding operation to increase a pixel frame, and a 3×3 window is used for maximum filtering, and the maximum filter can be represented as: (12) wherein represents an output image, represents pixel values of a 3x3 matrix centered at represents taking the maximum value;​ Then, the multi-channel high-frequency diffusion features subjected to the maximum processing are subjected to a sign function to generate an activity map, a 3×3 convolution layer is used to map the high-dimensional fusion features to an output of 128 channels, the threshold value of each pixel is calculated according to the local region of the output, and a mapping matrix is generated according to the adaptive threshold value, and the calculation process can be represented as: (13) (14) wherein, denotes a sign function, and denotes a different filtered image, denotes a mapping value of a convolution matrix, denotes a mapping matrix; Finally, the parameter 0 in the mapping matrix is replaced by the pixel corresponding to the infrared image, and the parameter 1 is replaced by the pixel corresponding to the visible light image, and then the fusion detail map is obtained through the convolution layer mapping ; S8, the multi-channel low-frequency diffusion features extracted by the detail adaptive denoising network are subjected to low-frequency feature fusion through the multi-channel low-frequency feature fusion network, first, the multi-channel low-frequency diffusion features are subjected to nearest neighbor up sampling and 1×1 convolution kernel combination processing, and then a series of convolution kernels with a convolution layer of 3×3 are used to map the low-dimensional information features to an output of 3 channels to obtain a fused base image; S9, the fused detail image dominated by high-frequency features obtained through the adaptive region consistency fusion network and the fused base image dominated by low-frequency features obtained through the multi-channel low-frequency feature fusion network are fused through the Concat strategy to reconstruct a fused image of infrared and visible light.

Images