Infrared and visible light image frequency domain diffusion fusion method

Abstract

The invention relates to an infrared and visible light image frequency domain diffusion fusion method, and belongs to the technical field of image processing. Comprising the following steps: inputting an infrared image and a visible light image into a potential space encoder to form original bimodal features; a Gaussian noise is randomly sampled, the Gaussian noise and the original bimodal features are input into a wavelet denoising network together to be decomposed into a low-frequency component and three high-frequency components, the bimodal high-frequency components are fused through FMM, and the bimodal low-frequency components are fused through FMM at the lowest scale; carrying out inverse discrete wavelet transform on the fused low-frequency component and high-frequency component to obtain estimated noise, and then carrying out noise removal on the estimated noise to obtain de-noising features; carrying out iterative optimization and feature updating to obtain fusion features; and inputting the fusion features into a potential space decoder to obtain a fusion image. The fusion quality of the infrared and visible light images can be comprehensively improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a frequency domain diffusion fusion method for infrared and visible light images. Background Art

[0002] Infrared sensors can effectively detect concealed thermal targets and exhibit excellent anti-interference capabilities, accurately distinguishing infrared targets from background areas even in complex environments. However, infrared imaging has significant limitations. Because it relies on thermal radiation, it struggles to accurately capture the target's geometric structure and surface texture details, resulting in generally low-resolution and detail-depleted images. In contrast, visible light sensors, based on the principle of light reflection imaging, can restore complex texture structures and rich spatial details in a scene. However, their image quality is susceptible to interference from environmental factors such as fluctuating light intensity, fog, haze, rain, and snow. To leverage the complementary advantages of these two modalities, infrared and visible light image fusion technology has emerged. By deeply fusing dual-modal features, this technology generates enhanced images that combine thermal target saliency with rich texture details. This not only improves target detection confidence, but also enhances environmental perception and optimizes scene representation. Currently, this fusion technology has been widely applied in key computer vision fields such as target tracking, person re-identification, and autonomous driving, significantly improving system robustness and recognition accuracy in complex scenes.

[0003] Traditional image fusion methods generally employ multi-scale transformations, sparse representations, optimization estimation, and hybrid models to address the fusion of infrared and visible light images. These methods rely on unified mathematical models for feature extraction and reconstruction. However, these inherent mathematical models cannot effectively capture the dynamic and changing intrinsic characteristics of the two image types, resulting in significant limitations in complex scene applications. Furthermore, traditional fusion methods often require manually designed fusion rules. This human bias significantly reduces the objectivity and reliability of the fusion output, further impacting subsequent objective analysis and decision-making, limiting the practical application of these fusion methods.

[0004] In recent years, deep learning-based image fusion methods have made significant progress, primarily due to their powerful feature extraction and representation capabilities. Currently, fusion methods based on CNNs and Transformers typically employ convolution operations and self-attention mechanisms to model local and global image features, improving fusion performance by enhancing feature representation capabilities. Image fusion methods based on generative adversarial networks (GANs) have become a highly representative deep learning paradigm in the field of image fusion. Within a generative adversarial framework, this method leverages adversarial training mechanisms to learn data distributions. GAN-based fusion techniques can preserve critical brightness information and structural details, resulting in high-quality fused images. However, despite these advantages, GAN-based models still face numerous challenges. For example, the model training process is unstable, the generated fusion results lack interpretability, and they are susceptible to mode collapse. These challenges often lead to inconsistent quality of fused images, limiting the widespread adoption of such methods in practical applications.

[0005] Recently, several studies have focused on applying diffusion models to image fusion, exploring various innovative strategies and achieving impressive results. However, despite the progress achieved, several key challenges and limitations remain. First, due to the difficulty in obtaining ground truth for image fusion, most existing methods rely heavily on diffusion priors to guide the fusion process. Currently, a unified diffusion fusion paradigm has not yet been established, making end-to-end training difficult and limiting the full potential of diffusion models in image fusion. Second, existing image fusion methods based on diffusion models generally overlook the critical role of frequency information in the fusion process. In reality, low-frequency signals in an image contain information about its overall structure, while high-frequency components can accurately represent edge details. Therefore, fully exploring and utilizing frequency features will significantly enhance the image's feature representation capabilities, thereby pushing image fusion performance to new heights.

[0006] In summary, there is an urgent need to construct a unified paradigm for frequency-domain generative diffusion fusion. This paradigm should be able to enhance the ability to represent deep features, effectively enhance useful information during the fusion process, and accurately suppress interference information, thereby comprehensively improving the fusion quality of infrared and visible light images. Summary of the Invention

[0007] To solve the above technical problems, the present invention provides a method for frequency domain diffusion fusion of infrared and visible light images. The technical solution of the present invention is as follows: A frequency domain diffusion fusion method for infrared and visible light images, wherein the frequency domain diffusion fusion method for infrared and visible light images is implemented using a pre-trained wavelet diffusion fusion network, wherein the wavelet diffusion fusion network includes a latent space encoder, a forward denoising network, a wavelet denoising network, and a latent space decoder connected in sequence; The infrared and visible light image frequency domain diffusion fusion method includes: S1: Input the infrared image and visible light image to be fused into the latent space encoder, and map them into a low-dimensional latent space through the latent space encoder to obtain the latent features of the infrared image and the visible light image, thus forming the original bimodal features. S2, randomly sample a Gaussian noise and input it together with the original bimodal features into the wavelet denoising network. The wavelet denoising network performs multi-scale discrete wavelet transform on the input bimodal features, decomposing them into a low-frequency component and three high-frequency components. Each scale of the high-frequency components of the bimodal features is fused through FMM, and the low-frequency components of the bimodal features are fused through FMM at the lowest scale. The fused low-frequency components and high-frequency components are subjected to inverse discrete wavelet transform to obtain the estimated noise, and then the estimated noise is removed to obtain the denoised features. S3, iterative optimization and feature update: The denoised features are combined with the original bimodal features to generate the input signal for the next time step and input it into the wavelet denoising network. The wavelet denoising network performs discrete wavelet transform, fusion, inverse discrete wavelet transform and noise removal until the feature distribution converges to obtain the fused features. S4, inputs the fused features into the latent space decoder to obtain the fused image.

[0008] Optionally, the wavelet denoising network consists of two parts: a wavelet encoder and a wavelet decoder. The wavelet encoder is used to perform multi-scale feature extraction and feature fusion processing on the input signal, and the wavelet decoder is used to perform noise estimation on the features obtained by the fusion processing of the wavelet encoder.

[0009] Optionally, the wavelet encoder includes a dual-modal downsampling module, each modal downsampling module includes three downsampling modules of different scales connected in sequence, the downsampling modules of the first two scales include a residual block, a DWT and an FMM, and the downsampling module of the third scale includes a residual block, a DWT and two FMMs; The S2 performs multi-scale discrete wavelet transform on the input bimodal features, decomposing them into a low-frequency component and three high-frequency components, and fuses each scale of the bimodal high-frequency components through FMM. When fusing the bimodal low-frequency components through FMM at the lowest scale, it includes: S21, the input bimodal features are respectively input into the first-scale downsampling module of the bimodal downsampling module, and are processed by the first-scale residual block and DWT in the first-scale downsampling module to generate high-frequency components and low-frequency components corresponding to the first scale; wherein, the high-frequency components of the bimodal features are input into the first-scale FMM for fusion, and the low-frequency components of the bimodal features are further passed to the second-scale downsampling module; S22, the low-frequency components entering the second-scale downsampling module are processed by the residual block and DWT in the second-scale downsampling module to generate high-frequency components and low-frequency components corresponding to the second scale; wherein the high-frequency components of the two modalities are input into the second-scale FMM for fusion, and the low-frequency components of the two modalities are further passed to the third-scale downsampling module; S23, the low-frequency components in the downsampling module of the third scale are processed by the residual block and DWT in the downsampling module of the third scale to generate high-frequency components and low-frequency components corresponding to the third scale; wherein, the high-frequency components of the bimodal are input to the first FMM of the third scale for fusion, and the low-frequency components of the bimodal are input to the second FMM of the third scale for fusion.

[0010] Optionally, the wavelet decoder includes three scale upsampling modules, each scale upsampling module includes a residual block and an IDWT connected in sequence; the wavelet decoder performs noise estimation on the features obtained by the wavelet encoder fusion processing, including: S24, after the low-frequency fusion features obtained by the second FMM fusion of the third scale enter the upsampling module of the third scale, the residual block of the third scale performs feature fusion processing on the low-frequency fusion features obtained by the second FMM fusion of the third scale, and then integrates the low-frequency features processed by the residual block of the third scale with the high-frequency features of the third scale obtained by the first FMM fusion of the third scale, completes feature reconstruction through the IDWT of the third scale, and finally outputs the fusion features of the third scale; S25, after the fused features of the third scale enter the upsampling module of the second scale, the residual block of the second scale performs feature extraction, channel alignment and optimization on the fused features of the third scale, and the features processed by the residual block of the second scale are fused with the high-frequency fused features of the second scale obtained by the FMM of the second scale, and feature reconstruction is achieved through IDWT, and finally the fused features of the second scale are output; S26, after the second-scale fusion feature enters the first-scale upsampling module, the first-scale residual block performs feature extraction, channel alignment, and optimization on the second-scale fusion feature. The features processed by the first-scale residual block are fused with the first-scale FMM to obtain the first-scale high-frequency fusion feature. Feature reconstruction is achieved through IDWT, and finally the first-scale fusion feature is output as the estimated noise.

[0011] Optionally, when fusing the high-frequency components of the two modalities, the FMM first combines the high-frequency components of the two modalities by a simple splicing operation, and then achieves preliminary fusion through a convolution operation; then, the features after preliminary fusion are subjected to average pooling and full connection processing in turn, and then the softmax function is used to assign corresponding weights to the high-frequency components of the two modalities; finally, the assigned weights are multiplied by the original high-frequency components of the two modalities and then added together, and the fused high-frequency features are output through a convolution operation.

[0012] Optionally, the residual block includes a normalization layer, an activation function layer, a convolution layer with a convolution kernel size of 3×3, an MHA layer, and a layernorm normalization layer; the number of channels of the three scales of the wavelet encoder and the wavelet decoder are 12, 24, 48 and 48, 24, 12 respectively; the activation function layer of each residual block is implemented by a Sigmod function.

[0013] Optionally, S3 includes: in the t-1th optimization and feature updating process, the denoising features obtained in the t-th iteration and the original bimodal features together constitute the t-1th input of the wavelet denoising network.

[0014] Optionally, the method further includes: When training the wavelet diffusion fusion network, for infrared training images and visible light training images, after extracting the original bimodal features of the infrared training images and visible light training images, Gaussian noise is gradually injected into the latent features of the infrared training images and visible light training images at time step sequence t = 1, 2, ..., T respectively through the forward denoising network to generate multiple noisy latent feature pairs. The noisy latent feature pairs and time step t are used to train the wavelet denoising network; When testing the wavelet diffusion fusion network, a Gaussian noise is randomly sampled and together with the original bimodal features, it constitutes the input of the initial stage of the wavelet denoising network, that is, T time steps. After one denoising, the obtained denoised features continue to be used as the input of the wavelet denoising network in T-1 time steps together with the original bimodal features until the iteration ends after T iterations.

[0015] Optionally, the loss function of the wavelet diffusion fusion network includes a fusion loss function and a diffusion loss function, which can be expressed as: (1); (2); (3); (4); (5); (6); in, represents the loss function of the wavelet diffusion fusion network; represents the fusion loss function; represents the diffusion loss function; Is a hyperparameter, representing the coefficient ratio of the fusion loss function to the diffusion loss function; represents the original bimodal features; t is the representation of the current time step; Represents the Gaussian noise added when the wavelet diffusion fusion network is forward noised; represents the noise estimated by the wavelet denoising network; represents the wavelet structural similarity loss function; represents the wavelet texture loss function; Represents the wavelet intensity loss function; 1. and are the coefficients of the wavelet structure similarity loss function, wavelet intensity loss function, and wavelet texture loss function; Represents structural similarity calculation; 、 and Represent the final fused image and the source images of the two modalities respectively; and Respectively represent the maximum operation and the L1 norm operation, represents the Sobel gradient operator, H and W Indicates the height and width of the image, mean Indicates the averaging operation; Represents the L2 norm operation.

[0016] All the above optional technical solutions can be combined arbitrarily, and the present invention does not provide detailed descriptions of the structures after each combination.

[0017] By means of the above solution, the beneficial effects of the present invention are as follows: By setting up a wavelet diffusion fusion network including a latent space encoder, a forward denoising network, a wavelet denoising network and a latent space decoder connected in sequence, the constructed wavelet diffusion fusion network can seamlessly integrate frequency representation, diffusion modeling and information fusion into a single process, realizing for the first time that information fusion is carried out directly in the frequency-based diffusion process, providing a more structured, interpretable and flexible fusion paradigm for information fusion.

[0018] By setting up a wavelet denoising network to perform multi-scale discrete wavelet transforms on the input bimodal features, decomposing them into one low-frequency component and three high-frequency components, an adaptive diffusion inference strategy specifically for the low-frequency components is proposed. This strategy enables accurate modeling of global information while independently preserving high-frequency edge details. This strategy not only strengthens feature representation but also significantly improves training stability and convergence efficiency. Furthermore, the wavelet denoising network uses wavelet transforms instead of traditional convolution and pooling operations, which can minimize information loss and effectively preserve fine-grained details, thereby generating higher-quality fused images.

[0019] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a schematic diagram of the composition structure of the wavelet diffusion fusion network in the present invention.

[0021] Figure 2 It is a schematic diagram of the module structure of the FMM in the present invention.

[0022] Figure 3 Schematic diagram of the structure of the residual block in the present invention.

[0023] Figure 4 This is a schematic diagram of the fused image comparison of the TNO dataset "Kaptein_1123". DETAILED DESCRIPTION

[0024] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0025] The frequency domain diffusion fusion method of infrared and visible light images provided by the embodiment of the present invention is implemented by using a pre-trained wavelet diffusion fusion network. Figure 1 As shown, the wavelet diffusion fusion network includes a latent space encoder, a forward denoising network, a wavelet denoising network and a latent space decoder connected in sequence; The frequency domain diffusion fusion method of infrared and visible light images includes the following steps S1 to S5: S1, the infrared image to be fused With visible light images Both are input into the latent space encoder (LatentEncoder), which maps the infrared image and the visible light image to a low-dimensional latent space through the latent space encoder to obtain the latent features of the infrared image and the visible light image, forming the original bimodal features .

[0026] S2, randomly sample a Gaussian noise X , and together with the original bimodal features The two models are jointly input into a wavelet denoising network, which performs multi-scale discrete wavelet transform on the input bimodal features, decomposing them into a low-frequency component (LL) and three high-frequency components (LH, HL, HH). Each scale of the high-frequency components of the bimodal features is fused through FMM, and the low-frequency components of the bimodal features are fused through FMM at the lowest scale. The fused low-frequency components and high-frequency components are subjected to inverse discrete wavelet transform to obtain estimated noise, which is then removed to obtain denoised features.

[0027] This step performs the reverse denoising process of the wavelet diffusion fusion network.

[0028] S3, iterative optimization and feature update: The denoised features are combined with the original bimodal features to generate the input signal for the next time step and input it into the wavelet denoising network. The wavelet denoising network performs discrete wavelet transform, fusion, inverse discrete wavelet transform and noise removal until the feature distribution converges to obtain the fused features.

[0029] S4, inputs the fused features into the latent space decoder to obtain the fused image.

[0030] It should be noted that when performing frequency domain diffusion fusion of infrared and visible light images through the above steps, it is necessary to pre-train the wavelet diffusion fusion network. When training the wavelet diffusion fusion network, for the infrared training images and visible light training images, after extracting the original bimodal features of the infrared training images and visible light training images, the forward denoising network is used to gradually inject Gaussian noise into the potential features of the infrared training images and visible light training images in the time step sequence t=1, 2,…, T, respectively, to generate multiple noisy potential feature pairs. When testing the wavelet diffusion fusion network, a Gaussian noise X is randomly sampled and combined with the original bimodal features. , together constitute the input of the initial stage of the wavelet denoising network, that is, the T time step. The denoised features obtained after one denoising continue to be used as the input of the wavelet denoising network in the T-1 time step together with the original bimodal features until the iteration ends after T iterations.

[0031] Specifically, the Gaussian noise injected through the forward noise adding network is expressed by formula (1): (1); In formula (1), represents the noise control factor, . represents the noisy latent feature pairs generated at time step t.

[0032] When removing the estimated noise, it is achieved through the following denoising formula: (2); In formula (2), is the noise scheduling factor, , , is the estimated noise of the wavelet denoising network, is a randomly selected Gaussian noise, where .

[0033] Specifically, when testing the wavelet diffusion fusion network, the embodiment of the present invention randomly selects a Gaussian noise starting from time step T. X and compare it with the original bimodal features The wavelet denoising network is used to estimate the noise at the current time step. , and then according to the denoising formula, we get the denoising feature after removing one step of noise In the subsequent iteration process, the relatively purer denoising features obtained in the previous step are , and the original bimodal features The wavelet denoising network of the next step is input together, and discrete wavelet transform, fusion, inverse discrete wavelet transform and noise removal are continued until t = 0, the characteristic distribution converges.

[0034] It should be noted that, in the embodiment of the present invention, when training the wavelet diffusion fusion network, the denoising and denoising processes are implemented through the forward denoising network and the wavelet denoising network; during testing, the noise is randomly injected and directly input into the wavelet denoising network for processing together with the original bimodal features.

[0035] In a specific embodiment, the wavelet denoising network consists of two parts: a wavelet encoder and a wavelet decoder. The wavelet encoder is used to perform multi-scale feature extraction and feature fusion processing on the input signal, and the wavelet decoder is used to perform noise estimation on the features obtained by the wavelet encoder fusion processing.

[0036] In a specific embodiment, the wavelet encoder includes a dual-modal downsampling module, each modal downsampling module includes three downsampling modules of different scales connected in sequence, the first two scale downsampling modules include a residual block (ResBlock), a DWT (discrete wavelet transform) and an FMM (frequency modulation module), and the third scale downsampling module includes a residual block, a DWT and two FMMs; The step S2 performs multi-scale discrete wavelet transform on the input bimodal features, decomposes them into a low-frequency component and three high-frequency components, and fuses each scale of the bimodal high-frequency components through FMM. When the bimodal low-frequency components are fused through FMM at the lowest scale, the steps S21 to S23 are included: S21, the input bimodal features are respectively input into the first-scale downsampling module of the bimodal downsampling module, and are processed by the first-scale residual block and DWT in the first-scale downsampling module to generate high-frequency components and low-frequency components corresponding to the first scale; wherein, the high-frequency components of the bimodal features are input into the first-scale FMM for weight distribution and fusion, and the low-frequency components of the bimodal features are further transmitted to the second-scale downsampling module; The FMM of the first scale outputs the high-frequency fusion features of the first scale and stores them in the high-frequency fusion list of the first scale.

[0037] S22, the low-frequency components entering the second-scale downsampling module are processed by the residual block and DWT (feature extraction) in the second-scale downsampling module to generate high-frequency components and low-frequency components corresponding to the second scale; the high-frequency components of the two modalities are input into the second-scale FMM for fusion, and the low-frequency components of the two modalities are further passed to the third-scale downsampling module; The FMM of the second scale outputs the high-frequency fusion features of the second scale and stores them in the high-frequency fusion list of the second scale.

[0038] S23, the low-frequency components in the downsampling module of the third scale are processed by the residual block and DWT in the downsampling module of the third scale to generate high-frequency components and low-frequency components corresponding to the third scale; wherein, the high-frequency components of the bimodal are input to the first FMM of the third scale for fusion, and the low-frequency components of the bimodal are input to the second FMM of the third scale for fusion.

[0039] The first FMM of the third scale outputs high-frequency fusion features of the third scale and stores them in the high-frequency fusion list of the third scale. The second FMM of the third scale outputs low-frequency fusion features and stores them in the low-frequency fusion list.

[0040] Specifically, during the training of the wavelet diffusion fusion network, the decomposition process of the wavelet encoder is expressed as: (3); In formula (3), and Representing infrared and visible light characteristics respectively t Time step l The low-frequency component of the scale is LL, and Representing infrared and visible light characteristics respectively t Time step l The three high-frequency components of the scale, namely {LH, HL, HH}, is a wavelet encoder, , .

[0041] Specifically, in l = 1, for the infrared and visible light feature pairs : (4); (5); in, represents the residual operation in each downsampling module.

[0042] In a specific embodiment, the wavelet decoder includes three scale upsampling modules, each scale upsampling module includes a residual block and an IDWT (inverse discrete wavelet transform) connected in sequence; the wavelet decoder performs noise estimation on the features obtained by the wavelet encoder fusion processing, including the following steps S24 to S26: S24, after the low-frequency fusion features obtained by the second FMM fusion of the third scale enter the upsampling module of the third scale, the residual block of the third scale performs feature fusion processing on the low-frequency fusion features obtained by the second FMM fusion of the third scale, and then integrates the low-frequency features processed by the residual block of the third scale with the high-frequency features of the third scale obtained by the first FMM fusion of the third scale, completes feature reconstruction through the IDWT of the third scale, and finally outputs the fusion features of the third scale; S25, after the fused features of the third scale enter the upsampling module of the second scale, the residual block of the second scale performs feature extraction, channel alignment and optimization on the fused features of the third scale, and the features processed by the residual block of the second scale are fused with the high-frequency fused features of the second scale obtained by the FMM of the second scale, and feature reconstruction is achieved through IDWT, and finally the fused features of the second scale are output; S26, after the second-scale fusion feature enters the first-scale upsampling module, the first-scale residual block performs feature extraction, channel alignment, and optimization on the second-scale fusion feature. The features processed by the first-scale residual block are fused with the first-scale FMM to obtain the first-scale high-frequency fusion feature. Feature reconstruction is achieved through IDWT, and finally the first-scale fusion feature is output as the estimated noise.

[0043] Specifically, the data processing process of IDWT is expressed as: (6); In formula (6), , express l The fusion features of the step l =0, That is , is the estimated noise of the wavelet denoising network.

[0044] In a specific embodiment, Figure 2 As shown, when the FMM fuses the high-frequency components of the two modes, it first uses a simple splicing operation to fuse the high-frequency components of the two modes. and Combined together, and then initially fused through convolution operation (Conv); then, the features after preliminary fusion are processed by average pooling (Average Pooling) and full connection (Full Connection), and then the softmax function is used to assign corresponding weights to the high-frequency components of the dual modality and Finally, the assigned weights are multiplied by the high-frequency components of the original dual-mode and then added together, and the fused high-frequency features are output through the convolution operation. i represents an infrared image, v represents a visible light image, t represents the current time step, where The fused high-frequency features are expressed as: ⊙ , f represents fusion, and ⊙ represents Hadamard operation.

[0045] It should be noted here that when the low-frequency component of the dual modality is input to the second FMM of the third scale for fusion, the processing of the low-frequency component of the dual modality is the same as Figure 2 The principles shown are the same and will not be repeated here.

[0046] In a specific embodiment, Figure 3 As shown in the figure, each residual block includes a normalized layer (Normalized), an activation function layer, a convolution layer (Conv) with a convolution kernel size of 3×3, an MHA (multi-head attention mechanism layer) and a layernorm normalization layer (LayerNorm); the number of channels of the three scales of the wavelet encoder and wavelet decoder are 12, 24, 48 and 48, 24, 12 respectively; the activation function layer of each residual block is implemented by the Sigmod function.

[0047] In a specific embodiment, S3 includes: in the t-1th optimization and feature update process, the denoising features obtained in the t-th iteration and the original bimodal features together constitute the input of the t-1 stage of the wavelet denoising network, until the preset number of iterations (T) is reached, the iteration is stopped, and the fusion feature is obtained.

[0048] In a specific embodiment, the loss function of the wavelet diffusion fusion network includes a fusion loss function and a diffusion loss function. The diffusion loss function is used to better constrain and train the wavelet denoising network, and the fusion loss function is used to train the fused image. The two loss functions together constrain the entire wavelet diffusion fusion network. The loss function of the wavelet diffusion fusion network is expressed as: (1); (2); (3); (4); (5); (6); in, represents the loss function of the wavelet diffusion fusion network; represents the fusion loss function; represents the diffusion loss function; Is a hyperparameter, representing the coefficient ratio of the fusion loss function to the diffusion loss function; represents the original bimodal features; t is the representation of the current time step; Represents the Gaussian noise added when the wavelet diffusion fusion network is forward noised; represents the noise estimated by the wavelet denoising network; represents the wavelet structural similarity loss function; represents the wavelet texture loss function; Represents the wavelet intensity loss function; 1. and are the coefficients of the wavelet structure similarity loss function, wavelet intensity loss function, and wavelet texture loss function; Represents structural similarity calculation; 、 and Represent the final fused image and the source images of the two modalities respectively; and Respectively represent the maximum operation and the L1 norm operation, represents the Sobel gradient operator, H and W Indicates the height and width of the image, mean Indicates the averaging operation; Represents the L2 norm operation. i represents an infrared image, v represents a visible light image, f Represents the final fused image.

[0049] Wavelet structural similarity loss function Used to adjust the structural similarity between the dual modal and fused images. Wavelet texture loss function Forces the network to carefully preserve the complex texture details of both modalities. Wavelet intensity loss function It is used to enable the network to effectively preserve key pixel intensity information. Used to control the loss between the fused image and the source image. Wavelet denoising network used to train wavelet diffusion fusion network. 、 and Used to control the balance between the three wavelet loss functions.

[0050] Furthermore, in order to verify the image fusion effect of the wavelet diffusion fusion network, the embodiment of the present invention also verifies the trained wavelet diffusion fusion network.

[0051] Specifically, in the test phase, 25 sets of training images were selected from the TNO dataset for test validation. The training images were cropped into 10,563 pairs of images of size 256×256 using a sliding window with a step size of 12. The grayscale values ​​of the cropped images were then normalized to the range of [-1, 1] for model training. At the same time, a learning rate of 1×10 −4 The Adam optimizer is used to update the training model. Batchsize and Epoch are set to 4 and 5 respectively. In terms of loss function, the hyperparameters 、 ,and They are set to 8, 2, and 14 respectively. The experimental platform uses Intel I9-10850K, 64 GB memory and NVIDIA GeForce GTX 3090 GPU.

[0052] Seven typical image fusion methods were selected for comparison, including LRRNet, CrossFuse, TarDAL, Dif-Fusion, Diff-IF, LFDT-Fusion, and DDFM. Furthermore, objective evaluation metrics included average gradient (AG), phase congruence (PC), spatial frequency (SF), visual information fidelity (VIF), entropy (EN), standard deviation (SD), multi-scale structural similarity (MS-SSIM), and quality assessment based on feature similarity (Qabf). Verification results are summarized in the following two aspects.

[0053] (1) Subjective evaluation. Figure 4 A schematic diagram of the subjective comparison results of the TNO dataset Kaptein_1123 is provided. Through comparison, it can be found that the method provided by the embodiment of the present invention (Ours) has three advantages. First, the fusion result can retain the high-brightness target information in the infrared image. For typical infrared targets, such as the pedestrian in the figure, the fusion result of the present invention has target features with higher brightness than other methods. Second, the fusion result can retain the texture details of the visible light image. For example, Figure 4 For representative details like the eaves in the image, the fusion results of the present invention are more distinct and clearer than those of other methods. Finally, the fusion results achieve higher contrast and better visual quality. Compared with the source image and other fusion results, the method provided by the embodiments of the present invention can better preserve prominent target features and rich scene details, resulting in a more balanced fusion result.

[0054] (2) Objective evaluation. Table 1 shows the objective comparison results of 25 training images of the TNO dataset. The optimal average and suboptimal average are marked with bold and underline respectively. It can be seen that the method provided by the embodiment of the present invention achieves the optimal average of the indicators AG, SF, VIF, EN, SD and Qabf, and the second average of the indicators PC and MS-SSIM. Objective experiments show that the method provided by the embodiment of the present invention has better fusion performance than other methods. Ranking first in AG, SD and VIF shows the significant ability of the present invention in terms of clarity and visual impact. Obtaining the optimal value in EN and SCD further confirms the ability of the present invention to pass rich information to the fusion output while maintaining a strong correlation with the source image. In addition, the optimal PC indicator and the suboptimal MS-SSIM and indicators further demonstrate the advantages of the present invention in maintaining edge and structural integrity. The present invention focuses on the extraction and rational use of frequency information, establishes information interaction between images of different modalities, different domains and different scales, and the acquired attention feature map is more focused on infrared target perception and visible texture detail representation.

[0055]

[0056] In summary, the present invention has the following characteristics: 1. This embodiment of the present invention establishes a unified framework that seamlessly integrates three key aspects: frequency representation, diffusion modeling, and information fusion, into a coherent and unified process. Notably, this is the first time that information fusion has been directly integrated into a frequency-based diffusion process, creating a more coherent, interpretable, and flexible information fusion paradigm.

[0057] 2. This embodiment of the present invention proposes an adaptive diffusion inference strategy specifically designed for low-frequency subbands. This strategy accurately models global structural information while independently preserving high-frequency edge details. This innovative strategy not only enhances feature representation capabilities but also significantly improves training stability and convergence efficiency.

[0058] 3. This embodiment of the present invention designs a wavelet denoising network based on the classic principle of frequency decomposition and reconstruction. This network uses wavelet decomposition and reconstruction to replace the repeated downsampling and upsampling operations in traditional frameworks, effectively reducing information loss, successfully preserving fine-grained details, and generating higher-quality fused output.

[0059] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A frequency domain diffusion fusion method for infrared and visible light images, characterized in that: The infrared and visible light image frequency domain diffusion fusion method is implemented using a pre-trained wavelet diffusion fusion network, which includes a latent space encoder, a forward denoising network, a wavelet denoising network and a latent space decoder connected in sequence; The infrared and visible light image frequency domain diffusion fusion method includes: S1: Input the infrared image and visible light image to be fused into the latent space encoder, and map them into a low-dimensional latent space through the latent space encoder to obtain the latent features of the infrared image and the visible light image, thus forming the original bimodal features. S2, randomly sample a Gaussian noise and input it together with the original bimodal features into the wavelet denoising network. The wavelet denoising network performs multi-scale discrete wavelet transform on the input bimodal features, decomposing them into a low-frequency component and three high-frequency components. Each scale of the high-frequency components of the bimodal features is fused through FMM, and the low-frequency components of the bimodal features are fused through FMM at the lowest scale. The fused low-frequency components and high-frequency components are subjected to inverse discrete wavelet transform to obtain the estimated noise, and then the estimated noise is removed to obtain the denoised features. S3, iterative optimization and feature update: The denoised features are combined with the original bimodal features to generate the input signal for the next time step and input it into the wavelet denoising network. The wavelet denoising network performs discrete wavelet transform, fusion, inverse discrete wavelet transform and noise removal until the feature distribution converges to obtain the fused features. S4, inputs the fused features into the latent space decoder to obtain the fused image.

2. The frequency domain diffusion fusion method of infrared and visible light images according to claim 1, characterized in that: The wavelet denoising network consists of two parts: a wavelet encoder and a wavelet decoder. The wavelet encoder is used to perform multi-scale feature extraction and feature fusion processing on the input signal, and the wavelet decoder is used to perform noise estimation on the features obtained by the fusion processing of the wavelet encoder.

3. The frequency domain diffusion fusion method of infrared and visible light images according to claim 2, characterized in that: The wavelet encoder includes a dual-modal downsampling module, each modal downsampling module includes three downsampling modules of different scales connected in sequence, the downsampling modules of the first two scales include a residual block, a DWT and an FMM, and the downsampling module of the third scale includes a residual block, a DWT and two FMMs; The S2 performs multi-scale discrete wavelet transform on the input bimodal features, decomposing them into a low-frequency component and three high-frequency components, and fuses each scale of the bimodal high-frequency components through FMM. When fusing the bimodal low-frequency components through FMM at the lowest scale, it includes: S21, the input bimodal features are respectively input into the first-scale downsampling module of the bimodal downsampling module, and are processed by the first-scale residual block and DWT in the first-scale downsampling module to generate high-frequency components and low-frequency components corresponding to the first scale; wherein, the high-frequency components of the bimodal features are input into the first-scale FMM for fusion, and the low-frequency components of the bimodal features are further passed to the second-scale downsampling module; S22, the low-frequency components entering the second-scale downsampling module are processed by the residual block and DWT in the second-scale downsampling module to generate high-frequency components and low-frequency components corresponding to the second scale; wherein the high-frequency components of the two modalities are input into the second-scale FMM for fusion, and the low-frequency components of the two modalities are further passed to the third-scale downsampling module; S23, the low-frequency components in the downsampling module of the third scale are processed by the residual block and DWT in the downsampling module of the third scale to generate high-frequency components and low-frequency components corresponding to the third scale; wherein, the high-frequency components of the bimodal are input to the first FMM of the third scale for fusion, and the low-frequency components of the bimodal are input to the second FMM of the third scale for fusion.

4. The frequency domain diffusion fusion method of infrared and visible light images according to claim 3, characterized in that: The wavelet decoder includes three scale upsampling modules, each scale upsampling module includes a residual block and an IDWT connected in sequence; The wavelet decoder performs noise estimation on the features obtained by the fusion processing of the wavelet encoder, including: S24, after the low-frequency fusion features obtained by the second FMM fusion of the third scale enter the upsampling module of the third scale, the residual block of the third scale performs feature fusion processing on the low-frequency fusion features obtained by the second FMM fusion of the third scale, and then integrates the low-frequency features processed by the residual block of the third scale with the high-frequency features of the third scale obtained by the first FMM fusion of the third scale, completes feature reconstruction through the IDWT of the third scale, and finally outputs the fusion features of the third scale; S25, after the fused features of the third scale enter the upsampling module of the second scale, the residual block of the second scale performs feature extraction, channel alignment and optimization on the fused features of the third scale, and the features processed by the residual block of the second scale are fused with the high-frequency fused features of the second scale obtained by the FMM of the second scale, and feature reconstruction is achieved through IDWT, and finally the fused features of the second scale are output; S26, after the second-scale fusion feature enters the first-scale upsampling module, the first-scale residual block performs feature extraction, channel alignment, and optimization on the second-scale fusion feature. The features processed by the first-scale residual block are fused with the first-scale FMM to obtain the first-scale high-frequency fusion feature. Feature reconstruction is achieved through IDWT, and finally the first-scale fusion feature is output as the estimated noise.

5. The frequency domain diffusion fusion method of infrared and visible light images according to claim 3, characterized in that: When fusing the high-frequency components of the two modalities, the FMM first combines the high-frequency components of the two modalities by a simple splicing operation, and then achieves preliminary fusion through convolution operation; then, the features after preliminary fusion are subjected to average pooling and full connection processing in turn, and then the softmax function is used to assign corresponding weights to the high-frequency components of the two modalities; finally, the assigned weights are multiplied by the original high-frequency components of the two modalities and then added together, and the fused high-frequency features are output through convolution operation.

6. The frequency domain diffusion fusion method of infrared and visible light images according to claim 3, characterized in that: The residual block includes a normalization layer, an activation function layer, a convolution layer with a convolution kernel size of 3×3, an MHA layer, and a layernorm normalization layer; the number of channels of the three scales of the wavelet encoder and wavelet decoder are 12, 24, 48 and 48, 24, 12 respectively; the activation function layer of each residual block is implemented by the Sigmoid function.

7. The frequency domain diffusion fusion method of infrared and visible light images according to claim 1, characterized in that: The S3 includes: in the t-1th optimization and feature updating process, the denoising features obtained in the t-th iteration and the original bimodal features together constitute the t-1th input of the wavelet denoising network.

8. The frequency domain diffusion fusion method of infrared and visible light images according to claim 1, characterized in that: The method further comprises: When training the wavelet diffusion fusion network, for infrared training images and visible light training images, after extracting the original bimodal features of the infrared training images and visible light training images, Gaussian noise is gradually injected into the latent features of the infrared training images and visible light training images at time step sequence t = 1, 2, ..., T respectively through the forward denoising network to generate multiple noisy latent feature pairs. The noisy latent feature pairs and time step t are used to train the wavelet denoising network; When testing the wavelet diffusion fusion network, a Gaussian noise is randomly sampled and together with the original bimodal features, it constitutes the input of the initial stage of the wavelet denoising network, that is, T time steps. After one denoising, the obtained denoised features continue to be used as the input of the wavelet denoising network in T-1 time steps together with the original bimodal features until the iteration ends after T iterations.

9. The frequency domain diffusion fusion method of infrared and visible light images according to claim 1, characterized in that: The loss function of the wavelet diffusion fusion network includes a fusion loss function and a diffusion loss function, which can be expressed as: （1）； （2）； （3）； （4）； （5）； （6）； in, represents the loss function of the wavelet diffusion fusion network; represents the fusion loss function; represents the diffusion loss function; Is a hyperparameter, representing the coefficient ratio of the fusion loss function to the diffusion loss function; represents the original bimodal features; t is the representation of the current time step; Represents the Gaussian noise added when the wavelet diffusion fusion network is forward noised; represents the noise estimated by the wavelet denoising network; represents the wavelet structural similarity loss function; represents the wavelet texture loss function; Represents the wavelet intensity loss function; 1. and are the coefficients of the wavelet structure similarity loss function, wavelet intensity loss function, and wavelet texture loss function; Represents structural similarity calculation; 、 and Represent the final fused image and the source images of the two modalities respectively; and Respectively represent the maximum operation and the L1 norm operation, represents the Sobel gradient operator, H and W Indicates the height and width of the image, mean Indicates the averaging operation; Represents the L2 norm operation.

Images