Denoising diffusion model driven texture enhanced infrared and visible light image fusion method and system

Abstract

The invention discloses a de-noising diffusion model driven texture enhanced infrared and visible light image fusion method and system, and relates to the technical field of image processing and computer vision. The method comprises the following steps: performing multi-scale space-time modeling on infrared and visible light images; enhancing high-frequency texture features of the infrared image through an edge detection convolutional network; based on a bidirectional multi-scale convolution module and a bidirectional attention fusion module, capturing a remote dependency relationship and encoding multi-scale information to ensure full utilization of global information and accurate capture of local details; and optimizing the structural fidelity and color intensity distribution of the fusion result based on an adaptive structural similarity loss function and multi-channel intensity and texture loss so as to improve the quality of the fused image. According to the method, the process of diffusion feature extraction is enhanced by fusing the de-noising diffusion model and the texture enhancement strategy. Meanwhile, a bidirectional multi-scale convolution technology and an attention mechanism are combined, so that comprehensive utilization of global information and accurate mastering of local details are ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and computer vision, and in particular to a method and system for fusion of texture-enhanced infrared and visible light images driven by a denoising diffusion model. Background Art

[0002] As an effective strategy to break through the limitations of a single sensor, infrared and visible light image fusion technology can provide us with a more comprehensive and accurate scene description based on the imaging characteristics of different sensors. Infrared sensors generate images by sensing the thermal radiation of objects. They are good at capturing clear target information in complex environments such as night and haze, but their images often lack rich texture details. In contrast, visible light sensors form images by capturing light reflected from objects, which can provide detailed structural and texture information, but their performance is limited in low light or occlusion conditions. Therefore, infrared and visible light image fusion technology can cleverly combine the advantages of these two sensors to provide high-quality image information for various complex environments.

[0003] Most traditional image fusion methods rely on mathematical transformation and rule design to achieve effective integration of multi-source images. These methods include multi-scale decomposition, subspace transformation, sparse representation and optimization algorithms. Among them, the multi-scale decomposition method fuses the image by decomposing it into details and approximation components of different scales. Typical techniques include wavelet transform and Laplace pyramid decomposition. However, these traditional methods have limitations in dealing with the unique characteristics of infrared and visible light images. It is difficult to take into account the thermal radiation characteristics of infrared images and the detailed texture of visible light images at the same time, resulting in incomplete image information after fusion. In addition, traditional methods are highly dependent on empirical parameters and are difficult to achieve adaptive adjustment in complex and changing scenes, which limits their practicality and scope of promotion.

[0004] With the rapid progress of deep learning technology, deep learning-based fusion methods have gradually become a new research hotspot. As an important model in the field of deep learning, convolutional neural network (CNN) has been widely used in image fusion tasks. By automatically learning high-level features of images, CNN overcomes the shortcomings of traditional methods that rely on manually designed features. However, CNN-based fusion methods still face challenges in color fidelity, detail retention, and artifact reduction, especially in complex scenes, where the effect may not be satisfactory.

[0005] Generative adversarial networks (GANs) have also been introduced into the field of image fusion. GANs can generate realistic fused images through adversarial training of generators and discriminators. However, the training process of GANs is complex and unstable, and has high requirements for data quality and parameter tuning. In addition, images generated by GANs may be accompanied by artifacts and lack sufficient interpretability, which to some extent hinders their promotion in practical applications.

[0006] Autoencoders extract image features through encoders, and then use decoders to achieve feature fusion and reconstruction. This type of method has certain advantages in feature extraction, but still needs to be improved in detail preservation and color restoration. At the same time, deep learning models usually require a large amount of labeled data for training, and it is expensive to obtain high-quality fusion data sets, which also limits its scope of application.

[0007] In summary, both traditional methods and deep learning fusion methods face many challenges in dealing with infrared and visible light image fusion tasks in complex scenes, including how to effectively balance different image features, improve the stability of fusion quality, enhance the interpretability of the model, and improve real-time processing capabilities. Summary of the invention

[0008] The purpose of the present invention is to propose a denoising diffusion model driven texture enhanced infrared and visible light image fusion method and system, which significantly improves the quality and visual effect of the fused image, while enhancing the structural fidelity and information integrity of the fusion result.

[0009] According to a first aspect of an embodiment of the present disclosure, a method for fusion of infrared and visible light images with texture enhancement driven by a denoising diffusion model is provided, comprising the following steps:

[0010] Multi-scale spatiotemporal modeling of infrared and visible light images;

[0011] Enhance high-frequency texture features of infrared images through edge detection convolutional networks;

[0012] Based on the bidirectional multi-scale convolution module and the bidirectional attention fusion module, it captures long-range dependencies and encodes multi-scale information to ensure full utilization of global information and accurate capture of local details;

[0013] Based on the adaptive structural similarity loss function, as well as multi-channel intensity and texture losses, the structural fidelity and color intensity distribution of the fusion results are optimized to improve the quality of the fused image.

[0014] According to a second aspect of an embodiment of the present disclosure, a denoising diffusion model driven texture enhanced infrared and visible light image fusion system is provided, comprising:

[0015] The spatiotemporal modeling module performs multi-scale spatiotemporal modeling of infrared and visible light images;

[0016] The enhancement module enhances the high-frequency texture features of infrared images through edge detection convolutional networks;

[0017] The image fusion module, based on the bidirectional multi-scale convolution module and the bidirectional attention fusion module, captures long-range dependencies and encodes multi-scale information to ensure full utilization of global information and accurate capture of local details;

[0018] The optimization module optimizes the structural fidelity and color intensity distribution of the fusion results to improve the quality of the fused image based on the adaptive structural similarity loss function and multi-channel intensity and texture losses.

[0019] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored and running on the memory, wherein when the processor executes the program, the denoising diffusion model-driven texture-enhanced infrared and visible light image fusion method is implemented.

[0020] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the denoising diffusion model-driven texture-enhanced infrared and visible light image fusion method is implemented.

[0021] Compared with the prior art, the above technical solution adopted by the present invention has the following advantages: the present invention inputs infrared and visible light images into a diffusion model for multi-scale spatiotemporal modeling, and uses an edge detection convolutional network to enhance the high-frequency texture features of the infrared image. These features are combined with the multi-scale spatiotemporal features extracted by the diffusion model, thereby significantly improving the quality of multi-source image fusion. Secondly, by using bidirectional multi-scale convolution and attention mechanisms, the long-range dependencies in the image are effectively captured, and the multi-scale information is accurately encoded, ensuring the seamless combination of global information and local details in the fusion process. In this way, the present invention can simultaneously retain the global structure and fine local details of the image during image fusion, further improving the overall quality of the fusion result. Finally, an adaptive structural similarity loss function and multi-channel intensity and gradient losses are designed to optimize the structural fidelity and color intensity distribution of the fusion result, thereby further improving the quality and visual effect of the fused image. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings in the specification, which constitute a part of the present application, are used to provide further understanding of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

[0023] Figure 1 The block diagram of the principle of fusion of texture-enhanced infrared and visible light images driven by denoising diffusion model;

[0024] Figure 2 Schematic diagram of the bidirectional multi-scale convolution module;

[0025] Figure 3Schematic diagram of the bidirectional attention fusion module;

[0026] Figure 4 This is a comparison chart of the fusion results in the daytime scene of the MSRS dataset numbered 00332D;

[0027] Figure 5 This is a comparison chart of the fusion results in the night scene of the MSRS dataset numbered 01016N;

[0028] Figure 6 This is a qualitative comparison diagram of the fusion results in the TNO dataset scenario;

[0029] Figure 7 Cumulative distribution curves for quantitative comparison of 25 pairs of images in the TNO dataset. DETAILED DESCRIPTION

[0030] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0031] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0032] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0033] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the methods and systems according to various embodiments of the present disclosure. It should be noted that each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of a code may include one or more executable instructions for implementing the logical functions specified in each embodiment. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, or they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the flowchart and / or block diagram, and the combination of boxes in the flowchart and / or block diagram can be implemented using a dedicated hardware-based system that performs a specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0034] Embodiment 1:

[0035] like Figure 1 As shown, this embodiment provides a denoising diffusion model driven texture enhanced infrared and visible light image fusion method, comprising the following steps:

[0036] Step 1: Multi-scale spatiotemporal modeling of infrared and visible light images:

[0037] Specifically, the denoising diffusion model generates high-quality images through two Markov decision processes (MDPs). In the forward chain, the data is perturbed into Gaussian noise, and the original structure is broken. The reverse chain reduces the noise by stepwise sampling to restore the original data. This noise removal process enables the denoising diffusion model to generate high-quality images that are highly close to the distribution of the training data. The forward and backward diffusion processes are briefly described below.

[0038] Forward diffusion process: by passing data sample x 0 ~q(x 0 ) injects noise to gradually disturb the data until the data is transformed into a simple prior distribution, that is, a standard Gaussian distribution. The diffusion process at each time t is the distribution x of the previous time t-1 Add Gaussian perturbation, that is:

[0039]

[0040] In the formula, the variance β t ∈(0,1) is the predefined noise scheduling parameter, which gradually increases with the increase of time steps. When the number of diffusion steps is large enough, the final obtained x t The original data distribution will be lost and degenerated into noise that obeys the standard Gaussian distribution.

[0041] Based on the Markov property and the chain rule, the forward chain can generate a series of random variables x 1 ,x 2 ···x T ; Decomposing the joint distribution of this random variable can obtain the conditional distribution of q(x t |x 0 ) analytical form. By means of parameter renormalization, let α t =1-β t and We can get:

[0042]

[0043] Thus, given any x 0 You can sample and transform the Gaussian vector ε~(0,1) to obtain the sample x at time step tt In addition, only Set it to a value close to 0 to obtain q(x T )≈N(x T ; 0,I).

[0044] Backward diffusion process: The reverse process can be regarded as a denoising operation, which uses the prior distribution to generate an unstructured noise vector and gradually removes the noise through a backward chain with learnable parameters. If the true distribution q(x t-1 |x t ), then the sample distribution q(x 0 ), however, the true distribution q(x t-1 |x t ) is difficult to obtain in practical applications. Based on this, the DDPM model constructs a parameterized neural network to approximate this distribution p θ (x t-1 |x t ),Right now:

[0045] p θ (x t-1 |x t )=N(x t-1 ;μ θ (x t ; t),∑ θ (x t ,t))

[0046] Among them, p θ (x t-1 |x t ) obeys the standard Gaussian distribution with mean μ θ (x t ,t) and variance ∑ θ (x t ,t) is parameterized by the network,∑ θ (x t ,t)=σ 2 I; Using the posterior distribution q(x t-1 |x t ,x 0 ) instead of the forward distribution q(x t-1 |x t ) can simplify the distribution calculation. Under the Markov assumption, the posterior distribution can be expressed as:

[0047]

[0048] in:

[0049]

[0050] The network parameters θ can be trained by minimizing the KL divergence (KLD) between the joint distributions of the forward and backward directions, and then optimized based on the following variational lower bound so that p θ (x t-1 |x t ) as close as possible to the posterior distribution q(x) t-1 |x t ,x 0 ):

[0051]

[0052] Among them, D KL (q(x T |x 0 )||p θ (x T )) and -log p θ (x 0 |x 1 ) does not depend on the model parameters and therefore does not affect the variational lower bound optimization. t-1 |x t ,x 0 ) and p θ (x t-1 |x t ), the KL divergence can be expressed as:

[0053]

[0054] The model optimization goal is to add the prediction to the current image x t The accumulated noise ∈ 0 To restore the original image, this process can be described by the following training objective function:

[0055]

[0056] First, based on the infrared and visible light images, a multi-channel noise image is constructed:

[0057]

[0058] in, It is multi-channel data, and the noise ε obeys the standard Gaussian distribution N~(0,1). The generated noise image is input into the diffusion feature extraction network to obtain the corresponding multi-scale features at different time steps:

[0059] {F t,i＝4 ,···,F t,i＝0 =F DDPM (I t )

[0060] Among them, F DDPM (·) represents the diffusion feature extraction network, F t,i represents the spatial multi-scale features extracted at time t, i∈(0,···,4) represents the spatial resolution of the multi-scale features extracted by the diffusion feature extraction network, which are 1 / 16, 1 / 8, 4 / 1 and 1 of the input respectively. When i=1, the feature spatial resolution is consistent with the input spatial resolution. The proposed model extracts spatial multi-scale features at multiple times and concatenates the obtained multi-spatial-temporal features to capture the effective information of multi-channel data under different noise levels, thereby providing richer and more reliable information for the fusion of spatial-temporal multi-scale diffusion features, and enhancing the feature correlation and robustness of infrared and visible multi-channel data.

[0061] Step 2: Enhance the high-frequency texture features of infrared images through edge detection convolutional networks:

[0062] Specifically, firstly, the important high-frequency information of the infrared image is extracted based on multi-layer convolution operations and nonlinear activation functions, namely:

[0063] I hf =I ir -Pool(ReLU(Conv(I ir )))

[0064] Among them, I ir represents an infrared image, Conv(·) represents a convolution operation, ReLU(·) is a nonlinear activation function, and Pool(·) represents an average pooling operation. The above operations further enhance the edge and texture details of the image while removing the low-frequency background. The proposed diffusion feature extraction model can capture information of different scales and provide global and local multi-scale representations for image fusion. However, infrared high-frequency features have a large spatial scale. In order to ensure that high-frequency features can be effectively fused with multimodal features, the high-frequency information is downsampled by the nearest interpolation method so that the high-frequency features I hf and diffusion characteristics F t,i With the same spatial resolution, that is:

[0065] I hf,aligned =Downsample(I hf ,scale)

[0066] In the formula, scale represents I hf With F t,i Then, the high-frequency features obtained under different spatial resolution conditions can be linked with the corresponding resolution features obtained under different time steps to obtain the following enhanced features:

[0067] F fusion,i =concat(F t,i ,Ihf,aligned )

[0068] In the formula, F fusion,i To enhance the obtained texture features, concat(·) represents linking by channel dimension. By directly integrating the high-frequency texture information of infrared images into multi-scale spatiotemporal features, the visual expression of the image can be effectively enhanced, while improving the ability to capture salient information of the target.

[0069] Step 3: Based on the bidirectional multi-scale convolution module and the bidirectional attention fusion module, long-range dependencies are captured and multi-scale information is encoded;

[0070] Specifically, Figure 2 As shown in the figure, the enhanced diffusion features are first convolved, and then the layer normalization operation is introduced to ensure the smoothness and consistency of feature processing. Subsequently, the dual-directional multi-scale convolution module (DMCM) inputs the obtained feature map into two parallel branches to extract the deep correlation information of the features. Branch F x In the figure, the feature map F is passed through a one-dimensional convolution layer with kernel sizes of 1×7, 1×11, and 1×21 to capture the horizontal depth feature information under different fields of view, namely:

[0071]

[0072] in, is the i-th branch convolution, and LN(·) represents layer normalization. Similarly, branch F y In the figure, the feature map F is passed through a one-dimensional convolution layer with kernel sizes of 7×1, 11×1, and 21×1 to capture the vertical depth feature information under different fields of view, namely:

[0073]

[0074] Through the above-mentioned bidirectional convolution strategy, the proposed DMCM can focus on both horizontal and vertical features at the same time, thereby mining the deep local details and overall structural information of the image, and thus obtain an effective expression of the multi-level features of the image.

[0075] Considering that high-level visual tasks require rich contextual information to obtain effective scene understanding, the present invention proposes Figure 3 The Bidirectional Attention Fusion Module (BAFM) module shown in Figure 1 converts the feature map F into a x and F y Convert to the following value (V x ,V y ), key (Kx ,K y ) and query (Q x ,Q y )vector:

[0076] K n =Reshape(Conv(F n )),V n =Reshape(Conv(F n )),Q n =Reshape(Conv(F n ))

[0077] Where n∈{x,y}, as well as Denote the key, value and query matrices respectively, Conv(·) and Reshape(·) correspond to the convolution layer with a kernel size of 3×3 and the reshaping operation respectively, H, W and C denote the input feature height, width and channel respectively. It should be noted that the present invention exchanges the two branch query matrices so that the two can obtain complementary information from each other to achieve feature interaction between branches, thereby enhancing the perception ability of each modality to each other and thus improving the comprehensive performance of the fusion result. Thus, the following attention map can be obtained:

[0078] A x =Softmax(Q y K x T ),A y =Softmax(Q x K y T )

[0079] Then, the value matrix is ​​multiplied by the obtained attention map to obtain the following context feature map with a global perspective:

[0080]

[0081] Based on the above fused feature maps, the spatial and channel attention layers are introduced respectively, and the following fused output can be obtained:

[0082] F out =SSE(F att )+CSE(F att )

[0083] The spatial and channel attention can be expressed as follows:

[0084] SSE(F att )=σ(Conv([GAP(F att ); GMP(F att )]))

[0085] CSE(F att )=σ(FC(ReLU(FC(GAP(F att )))))

[0086] Among them, σ represents the sigmoid activation function, FC represents the fully connected layer, GAP and GMP represent the global average pooling and global maximum pooling. The introduced spatial attention layer can capture the spatial relationship of the feature map to enhance the sensitivity of local information; while the channel attention layer focuses on the relationship between different channels to adaptively adjust the channel weights to improve the discriminative ability of the feature map. Based on the above, the obtained deep features are superimposed on the upper diffusion features until all the diffusion features are fused to generate a three-channel fused image.

[0087] Step 4: Based on the adaptive structural similarity loss function, as well as multi-channel intensity and gradient losses, the structural fidelity and color intensity distribution of the fusion results are optimized.

[0088] Specifically, the adaptive structural similarity loss is based on constraints such as light, contrast, and structure to ensure that the structure of the fusion result is highly consistent with the source image. Specifically, brightness is measured by mean difference, contrast is measured by variance, and structure is characterized by covariance to represent the structural similarity between the source image and the fusion result. Therefore, the following adaptive structural similarity loss function L is defined: ssim :

[0089] L ssim =W ir ×(1-ssim(I f ,I ir ))+W vis ×(1-ssim(I f ,I vis ))

[0090] Among them, ssim(·) represents the similarity index, W ir and W vis is the adaptive structural similarity loss weight, which can be expressed as follows:

[0091]

[0092] Among them, μ x and μ y represents the average brightness of the image, σ x and σ y represents the image contrast, σ xy is the covariance, C 1 and C 2 represents a constant used to stabilize the calculation, and Represents the mean gradient of infrared and visible light images. Weight Wir and W vis Normalize the weights to balance the contribution of the structural similarity loss between infrared and visible images.

[0093] In order to keep the fused image with the global optimal brightness, the following multi-channel intensity loss L is introduced: I :

[0094]

[0095] Among them, max(·) represents the maximum value operation. This loss function improves the global brightness visual effect of the fusion result by minimizing the pixel-level intensity difference between the fusion result and the source image.

[0096] In order to maintain the details and texture information of the fused image, the following multi-channel gradient loss L is introduced G :

[0097]

[0098] Among them, ▽I ir , ▽I vis And ▽I f,i are the gradients of the infrared, visible light, and fused image i-th channels respectively. By minimizing the gradient difference between the fused image and the source image in each channel to maintain the image details and texture information, the texture details and layering of the source image can be accurately represented.

[0099] Based on the above, the total loss can be expressed as follows:

[0100] L f =λL G +βL I +γL ssim

[0101] Among them, λ,β, γ is a weight parameter used to balance the contribution of each loss term.

[0102] This method strengthens the diffusion feature extraction process by fusing the denoising diffusion model with the texture enhancement strategy. At the same time, it combines the bidirectional multi-scale convolution technology and the attention mechanism to ensure the comprehensive utilization of global information and the accurate grasp of local details. In addition, by introducing the adaptive structural similarity loss and multi-channel loss function, the quality of the fused image is further optimized, thereby improving the structural fidelity and information integrity of the fusion result.

[0103] Embodiment 2:

[0104] This embodiment provides a denoising diffusion model driven texture enhanced infrared and visible light image fusion system, including:

[0105] The spatiotemporal modeling module performs multi-scale spatiotemporal modeling of infrared and visible light images;

[0106] The enhancement module enhances the high-frequency texture features of infrared images through edge detection convolutional networks;

[0107] Image fusion module, based on bidirectional multi-scale convolution module and bidirectional attention fusion module, captures long-range dependencies and encodes multi-scale information;

[0108] The optimization module optimizes the structural fidelity and color intensity distribution of the fusion results based on the adaptive structural similarity loss function, as well as multi-channel intensity and gradient losses.

[0109] Embodiment three:

[0110] An electronic device includes a memory, a processor, and a computer program stored and running on the memory, wherein the processor implements the above-mentioned denoising diffusion model driven texture enhanced infrared and visible light image fusion method when executing the program, including:

[0111] Multi-scale spatiotemporal modeling of infrared and visible light images;

[0112] Enhance high-frequency texture features of infrared images through edge detection convolutional networks;

[0113] Based on the bidirectional multi-scale convolution module and the bidirectional attention fusion module, it captures long-range dependencies and encodes multi-scale information;

[0114] Based on the adaptive structural similarity loss function, as well as multi-channel intensity and gradient losses, the structural fidelity and color intensity distribution of the fusion results are optimized.

[0115] Embodiment 4:

[0116] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned denoising diffusion model driven texture enhanced infrared and visible light image fusion method, comprising:

[0117] Multi-scale spatiotemporal modeling of infrared and visible light images;

[0118] Enhance high-frequency texture features of infrared images through edge detection convolutional networks;

[0119] Based on the bidirectional multi-scale convolution module and the bidirectional attention fusion module, it captures long-range dependencies and encodes multi-scale information;

[0120] Based on the adaptive structural similarity loss function, as well as multi-channel intensity and gradient losses, the structural fidelity and color intensity distribution of the fusion results are optimized.

[0121] In order to specifically describe the implementation effect of the present invention, the experimental conditions are set: in the diffusion model training stage, the resolution of the infrared and visible light images is cropped to 160×160 and input into the constructed DDPM network for fusion training. The resolution of the feature fusion infrared and visible light images is the original resolution, the model training is based on the Adam optimizer, the Epoch is set to 300, the learning rate is 0.0001, and the batch size is 12.

[0122] Experiment 1: Qualitative performance comparison on MSRS dataset. The fusion results of the proposed method and the comparison method are shown in Figure 2. Figure 4 and Figure 5 As shown in the figure, the proposed method and Dif-Fusion algorithm can effectively maintain good color fidelity and retain detailed texture information, while the other methods have different degrees of color distortion and detail blurring. Specifically, GTF, TIF and DenseFuse fail to fully extract multimodal features, resulting in the failure to effectively fuse the detail information of the visible light image, making the fusion result dark and the details lost. The fusion results obtained by FusionGAN and U2Fusion show an unbalanced saturation phenomenon, indicating that these methods lack targeted information weight adjustment when processing cross-modal images, resulting in color imbalance. In addition, artifacts appear at the edges of objects in the results obtained by Dif-Fusion, indicating that its strategy of simply superimposing features has great limitations. In contrast, the proposed method can effectively extract and fuse multi-scale detail information based on DMCM, while BAFM weightedly fuses multimodal features to avoid edge artifacts.

[0123] Experiment 2: Quantitative performance comparison experiment, as shown in Table 1; it can be seen that the present invention has achieved the highest values ​​in MI, VIF and SCD indicators, which means that the fusion result better retains the information characteristics of the source image and is more in line with the perception of human vision. The highest value of the Qabf indicator shows excellent performance in retaining edge information, ensuring the integrity of edge details of the fused image. The highest value of SD shows that the fusion result has an advantage in contrast, further enhancing the visual presentation of the image. Although the present invention is slightly inferior to Dif-Fusion in the SF indicator, this can be attributed to the application of adaptive structural similarity loss, which effectively removes some noise points and makes the fused image smoother.

[0124] Table 1 Fusion quality evaluation based on 300 pairs of images in the MSRS dataset

[0125]

[0126] Experiment 3: Qualitative performance comparison on TNO dataset. The fusion results of the proposed method and the comparison method are shown in Figure 3. Figure 6 As shown in the figure; the results of the generalization experiment based on 25 pairs of images in TNO are as follows Figure 6 As shown in the figure. It can be seen from the box-marked area that, except for the proposed method and Dif-Fusion, the background areas of the fusion results obtained by the other methods all have serious spectral pollution and edge artifacts, especially TIF, DenseFuse, FusionGAN and U2Fusion. This can be attributed to the fact that the above methods fail to effectively process the feature information of different modalities, resulting in interference in the information fusion process. Although the Dif-Fusion model can retain the source image information, it fails to fully fuse the edge information, resulting in the direct superposition of the bright area features in the visible light image to the thermal salient target, resulting in more obvious edge artifacts. The proposed method is based on bidirectional multi-scale convolution and bidirectional attention fusion modules, which can highly realistically maintain the background details while highlighting the infrared salient target, thereby effectively avoiding the generation of spectral pollution and artifacts.

[0127] Experiment 4: Quantitative performance comparison of TNO dataset, such as Figure 7 As shown; it can be seen from the cumulative distribution curve that the present invention has achieved better values ​​in evaluation indicators such as SD, MI, VIF and Qabf, indicating that it can effectively retain the key information of the source image in complex scenes and maintain a good balance in color contrast, so it has higher robustness and better generalization ability.

[0128] Those skilled in the art should understand that the modules or steps of the present disclosure can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present disclosure is not limited to any specific combination of hardware and software.

[0129] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0130] Although the above describes the specific implementation methods of the present disclosure in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present disclosure. Technical personnel in the relevant field should understand that on the basis of the technical solution of the present disclosure, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present disclosure.

Claims

1. A denoising diffusion model driven texture enhanced infrared and visible light image fusion method, characterized in that: include: Multi-scale spatiotemporal modeling of infrared and visible light images; Enhance high-frequency texture features of infrared images through edge detection convolutional networks; Based on the bidirectional multi-scale convolution module and the bidirectional attention fusion module, it captures long-range dependencies and encodes multi-scale information; Based on the adaptive structural similarity loss function, as well as multi-channel intensity and gradient losses, the structural fidelity and color intensity distribution of the fusion results are optimized.

2. According to claim 1, the denoising diffusion model driven texture enhanced infrared and visible light image fusion method is characterized in that: Multi-scale spatiotemporal modeling of infrared and visible light images is performed as follows: Based on infrared and visible light images, construct a multi-channel noise image: Among them, α t are parameters related to the predefined noise scheduling parameters, It is multi-channel data, and the noise ε obeys the standard Gaussian distribution N~(0,1); the multi-channel noise image is input into the diffusion feature extraction network to obtain the corresponding multi-scale features at different time steps: {F t,i＝4 ,···,F t,i＝0 }=F DDPM (I t ) Among them, F DDPM (·) represents the diffusion feature extraction network, F t,i represents the spatial multi-scale features extracted at time t, and i∈(0,···,4) represents the spatial resolution of the multi-scale features extracted by the diffusion feature extraction network.

3. The denoising diffusion model driven texture enhanced infrared and visible light image fusion method according to claim 1 or 2, characterized in that: The high-frequency texture features of infrared images are enhanced through edge detection convolutional networks, specifically: I hf =I ir -Pool(ReLU(Conv(I ir ))) Among them, I ir represents an infrared image, Conv(·) represents a convolution operation, ReLU(·) is a nonlinear activation function, and Pool(·) represents an average pooling operation; the high-frequency information is downsampled by the nearest interpolation method so that the high-frequency feature I hf and diffusion characteristics F t,i With the same spatial resolution, that is: I hf,aligned =Downsample(I hf ,scale) In the formula, scale represents I hf With F t,i The resolution scaling factor between them; then the high-frequency features obtained under different spatial resolution conditions are linked with the corresponding resolution features obtained under different time steps to obtain the following enhanced features: F fusion,i =concat(F t,i ,I hf,aligned ) In the formula, F fusion,i To enhance the obtained texture features, concat(·) represents concatenation along the channel dimension.

4. The denoising diffusion model driven texture enhanced infrared and visible light image fusion method according to claim 3, characterized in that: The internal implementation of the bidirectional multi-scale convolution module is shown as follows: The enhanced features are convolved and layer normalization is introduced to ensure the smoothness and consistency of feature map processing. Subsequently, the resulting feature map is input into two parallel branches to extract deep feature correlation information. Branch F x In , the feature map F passes through a one-dimensional convolution layer with convolution kernels of different sizes to capture the horizontal depth feature information under different fields of view, namely: in, is the i-th layer branch convolution, LN(·) represents layer normalization; Branch F y In , the feature map F passes through a one-dimensional convolution layer with convolution kernels of different sizes to capture the vertical depth feature information under different fields of view, namely:

5. The denoising diffusion model driven texture enhanced infrared and visible light image fusion method according to claim 1, characterized in that: The internal implementation of the bidirectional attention fusion module is as follows: The projection function including convolution and reshaping operations transforms the feature map F x and F y Convert to the following value (V x ,V y ), key (K x ,K y ) and query (Q x ,Q y )vector: K n =Reshape(Conv(F n )),V n =Reshape(Conv(F n )),Q n =Reshape(Conv(F n )) Where n∈{x,y}, as well as denote the key, value, and query matrices, respectively. Conv(·) and Reshape(·) correspond to the convolutional layer and reshape operation, respectively. H, W, and C denote the input feature height, width, and channels, respectively. The query matrices of the two branches are exchanged so that they can obtain complementary information from each other to achieve feature interaction between branches, and the following attention map is obtained: A x =Softmax(Q y K x T ),A y =Softmax(Q x K y T ) Then, the value matrix is ​​multiplied by the obtained attention map to obtain the following context feature map with a global perspective: Based on the above fused feature maps, spatial and channel attention layers are introduced respectively to obtain the following fused output: F out =SSE(F att )+CSE(F att ) The spatial and channel attention are expressed as follows: SSE(F att )=σ(Conv([GAP(F att );GMP(F att )])) CSE(F att )=σ(FC(ReLU(FC(GAP(F att ))))) Among them, σ represents the sigmoid activation function, FC represents the fully connected layer, GAP and GMP represent global average pooling and global maximum pooling.

6. The denoising diffusion model driven texture enhanced infrared and visible light image fusion method according to claim 1, characterized in that: Adaptive structural similarity loss function L ssim for: L ssim =W ir ×(1-ssim(I f ,I ir ))+W vis ×(1-ssim(I f ,I vis )) Among them, I f is the fused image, I ir is the infrared image, I vis is a visible light image; ssim(·) represents the similarity index, W ir and W vis are the adaptive structural similarity loss weights, which are expressed as follows: Among them, μ x and μ y represents the average brightness of the image, σ x and σ y represents the image contrast, σ xy is the covariance, C1 and C2 are constants used to stabilize the calculation, and Represents the mean gradient of infrared and visible light images; weight W ir and W vis Normalize the weights to balance the contribution of the structural similarity loss between infrared and visible images; In order to keep the fused image with the global optimal brightness, the following multi-channel intensity loss L is introduced: I : Among them, max(·) represents the maximum value operation; In order to maintain the details and texture information of the fused image, the following multi-channel gradient loss L is introduced G : Among them, ▽I ir , ▽I vis And ▽I f,i They are infrared, visible light and fusion image. i The gradient of each channel; Based on the above, the total loss is expressed as follows: L f =λL G +βL I +γL ssim Among them, λ, β, γ are weight parameters used to balance the contribution of each loss term.

7. A denoising diffusion model driven texture enhanced infrared and visible light image fusion system, characterized in that: include: The spatiotemporal modeling module performs multi-scale spatiotemporal modeling of infrared and visible light images; The enhancement module enhances the high-frequency texture features of infrared images through edge detection convolutional networks; Image fusion module, based on bidirectional multi-scale convolution module and bidirectional attention fusion module, captures long-range dependencies and encodes multi-scale information; The optimization module optimizes the structural fidelity and color intensity distribution of the fusion results based on the adaptive structural similarity loss function, as well as multi-channel intensity and gradient losses.

8. An electronic device comprising a memory, a processor and a computer program stored and running on the memory, wherein when the processor executes the program, the denoising diffusion model driven texture enhanced infrared and visible light image fusion method described in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the denoising diffusion model driven texture enhanced infrared and visible light image fusion method as described in any one of claims 1 to 6.

Images