An infrared image stripe removal method based on knowledge embedding diffusion model

Abstract

The application discloses an infrared image stripe removal method based on a knowledge embedding diffusion model, and comprises the following steps: S1, acquiring an infrared image with stripe noise to be processed; S2, inputting the infrared image with stripe noise into a trained stripe removal model to output a corresponding stripe-removed infrared image; when training the stripe removal model, the parameters of a denoising network are jointly optimized based on a noise loss function and a prior loss function with stripe noise direction prior introduced; and S3, outputting the output stripe-removed infrared image as a stripe removal result of the infrared image with stripe noise to be processed. The application introduces a directional wavelet convolution module as the knowledge prior of the model on the basis of the diffusion model. The module can fuse the stripe direction prior and semantic information, supplement the knowledge prior loss function of the calculation model, and directionally constrain the generation direction of the diffusion model, so that a composite infrared stripe removal mode driven by knowledge and data is formed.

Description

Technical Field

[0001] This invention relates to the fields of infrared imaging and artificial intelligence big data technology, specifically to a method for removing stripes from infrared images based on a knowledge embedding diffusion model. Background Technology

[0002] Infrared imaging (IR) technology, with its outstanding advantages such as all-weather operation, high concealment, and strong anti-interference capabilities, has important applications in military guidance, aerospace, remote sensing, and security monitoring. However, due to limitations in the manufacturing level and processing precision of infrared focal plane arrays (IRFPAs), the actual response of each detection unit in an infrared imaging system is often not completely consistent. This results in the presence of many non-uniform stripe noises in the output images, even under uniform radiation, known as fixed pattern noise (FPN) of the infrared focal plane array. The presence of FPN severely affects the visual quality of the output image, greatly reduces the temperature resolution and detection response capability of the infrared imaging system, and ultimately restricts the further application of infrared imaging technology. Therefore, it is necessary to correct the non-uniform stripe noise in the acquired infrared detection images to better facilitate subsequent image target recognition tasks.

[0003] Currently, research on infrared image stripe removal algorithms can be broadly categorized into two types: 1) methods based on traditional model optimization; and 2) methods based on deep learning. For methods in category 1), such as filtering methods, energy constraint minimization methods, and response statistics methods, the core principle is to use iterative optimization algorithms across single or multiple frames based on the visual differences between the image scene and stripe noise. These algorithms employ manually designed feature extraction operators to separate stripe noise from the image scene. While these algorithms are simple in design, they rely excessively on manually designed coarse features to describe image attributes, resulting in poor adaptability and difficulty in achieving acceptable correction effects for stripe noise of different intensities and types simultaneously. For methods in category 2), the core principle is to utilize the powerful feature representation capabilities of convolutional neural networks (CNNs) to learn from large amounts of sample data and extract the spatiotemporal distribution features of stripe noise for removal. In recent years, some methods have also been combined with deep generative models based on GAN networks to achieve self-supervised or unsupervised learning effects. Although deep learning-based stripe removal methods have demonstrated strong feature learning and modeling capabilities, they still have some insurmountable shortcomings. For example, CNN-based methods are limited in their prediction accuracy for complex semantic information due to the limited receptive field size of the convolutional kernel, resulting in the model being unable to completely distinguish between stripe noise and background texture. On the other hand, various generative models based on GANs face challenges such as mode collapse, model optimization difficulties, and gradient vanishing, all of which have an adverse effect on their task of removing stripes from infrared images.

[0004] In recent years, a novel architecture called the diffusion model in deep generative models has demonstrated significant application potential in various computer vision tasks. The applicant found that, compared to GAN models, the diffusion model can generate more realistic details and textures, output better image quality, and avoid the intractable problems of training instability and model optimization difficulties encountered by GAN models. The diffusion model employs a progressive refinement feature learning strategy, simulating the feature distribution of the target image domain through iterative sampling, which is consistent with the physical process of infrared image stripe removal. Therefore, designing an infrared image stripe removal method based on the diffusion model is a pressing technical problem to be solved. Summary of the Invention

[0005] To address the shortcomings of the existing technologies, the technical problem to be solved by this invention is: how to provide an infrared image stripe removal method based on a knowledge-embedded diffusion model. This method introduces a directional wavelet convolution module as a prior knowledge of the model, which can fuse the stripe direction prior with semantic information, supplement the calculation of the model's prior knowledge loss function, and use directional constraints to guide the generation direction of the diffusion model, forming a composite infrared stripe removal mode driven by both knowledge and data, thereby further improving the model's generalization performance.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0007] A method for removing infrared image stripes based on a knowledge embedding diffusion model includes:

[0008] S1: Acquire the infrared image with striped noise to be processed;

[0009] S2: Input the striped infrared image with noise to be processed into the trained stripe removal model, and output the corresponding striped infrared image;

[0010] The stripe removal model is built upon a latent diffusion model, and the training steps are as follows:

[0011] S201: Obtain the striped, noisy infrared image y and the corresponding clear infrared image x as training samples;

[0012] S202: Perform conditional encoding on the infrared image y with striped noise to obtain the conditional feature map c; encode the clear infrared image x to obtain the initial feature vector z0 of the latent space;

[0013] S203: For the initial eigenvector z0 in the latent space, standard Gaussian noise ε is continuously added at T discrete time nodes. t ~N(0,I), thus obtaining the latent eigenvector z at time t=T. T N(·) represents a Gaussian distribution, and I represents the identity matrix;

[0014] S204: Using T trained denoising networks combined with conditional feature maps c, sequentially process the latent feature vector z starting at time t = T. t Perform back-diffusion denoising until t=0 to obtain the latent feature estimate.

[0015] S205: Estimation of latent features via decoder The image was restored to its original state, resulting in a destriped infrared image.

[0016] S206: Standard Gaussian noise ε added at each time point tThe predicted noise ε output by the denoising network at the corresponding time step θ (z t The difference between z, t, and c is used to calculate the noise loss function; t This represents the intermediate latent eigenvector at time t;

[0017] S207: Based on clear infrared image x and destriated infrared image Calculate the prior loss function by introducing directional priors to fringe noise;

[0018] S208: Calculate the total loss function using the noise loss function and the prior loss function, and then optimize the parameters of the denoising network in reverse.

[0019] S209: Repeat steps S201 to S208 to iteratively train the denoising network until it converges or reaches the preset number of iterations;

[0020] After training, the stripe removal model undergoes the following processing steps:

[0021] S211: Encode the infrared image with striped noise to be processed to obtain the latent feature vectors at each time point;

[0022] S212: Perform conditional encoding on the infrared image with striped noise to be processed to obtain a conditional feature map;

[0023] S213: By combining the trained T denoising networks with the conditional feature map, the latent feature vectors starting from time t=T are sequentially back-diffusion denoised until time t=0, and the latent feature estimates are obtained.

[0024] S214: The latent feature estimation is recovered through the decoder to obtain the destriped infrared image;

[0025] S3: Output the destriped infrared image as the stripe removal result of the infrared image with striped noise to be processed.

[0026] Preferably, in steps S202 and S211, encoding is performed using a VAE variational autoencoder.

[0027] Preferably, in steps S202 and S212, the infrared image with striped noise is conditionally encoded using a conditional encoder.

[0028] Preferably, in step S203, T discrete time points are sampled, and standard Gaussian noise ε is randomly generated at each time point. t Starting from time t=0, the standard Gaussian noise ε corresponding to each time point is sequentially recorded. t Layer by layer, these vectors are added to the latent space vector z0 until time t = T, generating T intermediate latent feature vectors z1 to z2. T .

[0029] Preferably, in step S204, T denoising networks are set up, each corresponding to one of the T time nodes; at time t=T, the first denoising network decomposes the intermediate latent feature vector z. T The conditional feature map c is used as input for backdiffusion denoising, and the corresponding latent feature estimate is output. Subsequent denoising networks sequentially use the latent feature estimate and conditional feature map c output by the previous denoising network as input for backdiffusion denoising until time t=0, finally obtaining the latent feature estimate.

[0030] Preferably, in steps S205 and S214, recovery is performed using a VAE decoder.

[0031] Preferably, in step S206, the formula for the noise loss function is expressed as follows:

[0032] L noise =||ε t -ε θ (z t ,t,c)|| 2 ;

[0033] In the formula: L noise Indicates noise loss; ε t ε represents the added Gaussian noise; θ (z t (t,c) represents the predicted noise output of the denoising network, z t Let represent the intermediate latent feature vector at time t, and c represent the conditional feature map.

[0034] Preferably, in step S207, the prior loss function is calculated through the following steps:

[0035] S2071: The infrared clear image x and the destriped infrared image x are processed using a directional wavelet convolution module, respectively. Perform wavelet decomposition at multiple scales to obtain multi-scale wavelet decomposition results in different directions;

[0036] S2072: Extract the sharp infrared image x and the destriped infrared image. Wavelet vertical components x at various scales HL and

[0037] S2073: Using wavelet vertical component x HL and Calculate the prior loss function;

[0038] The formula is expressed as:

[0039]

[0040] In the formula: L prior This is a priori loss.

[0041] Preferably, in S2071, the directional wavelet convolution module includes four convolutional filters, which are defined as follows:

[0042]

[0043] In the formula: f LL It is a low-pass filter, corresponding to the low-frequency information of the input image; f LH f HL and f HH These are three high-pass filters, corresponding to the high-frequency information in the horizontal, vertical, and diagonal directions of the input image, respectively;

[0044] For a given infrared image I, the multi-scale wavelet decomposition results in different directions are obtained using a convolution filter, and their expression is:

[0045]

[0046] In the formula: Conv(·) represents the convolution operation, corresponding to 4 discrete wavelet convolutions, which are used to further decompose the low-frequency components at the current (i-1) scale into 4 components at the i scale; These represent the output results of the infrared image at the i-th scale in the wavelet low-frequency, horizontal, vertical, and diagonal directions, respectively.

[0047] Preferably, in step S208, the formula for calculating the total loss function is as follows:

[0048] L total =L noise +λL prior ;

[0049] In the formula: L total Let L represent the total loss function; λ is an adjustable hyperparameter; L prior For prior loss; L noise This indicates noise loss.

[0050] Compared with existing technologies, the infrared image stripe removal method based on the knowledge embedding diffusion model in this invention has the following advantages:

[0051] This invention creatively proposes a novel network model framework for infrared image stripe removal that combines prior knowledge with a diffusion model. First, by incorporating the spatial distribution characteristics of stripe noise into the diffusion model for constraint, and utilizing the progressive iterative sampling method of the diffusion model, it can generate better destriped output images. To our knowledge, this is the first infrared image destriping method based on a diffusion model. Second, this invention introduces a directional wavelet convolution module (DWCM) as the model's prior knowledge. DWCM can fuse stripe direction prior with semantic information, supplementing the calculation of the model's prior knowledge loss function. The directional constraint guides the generation direction of the diffusion model, forming a composite infrared destriping mode driven by both knowledge and data, thereby further improving the generalization performance of the stripe removal model (KEDM). Finally, to improve the model's training speed and reduce parameter complexity, this invention also introduces an improved latent diffusion model. By encoding and decoding the input and output images separately in a latent feature space, it can generate sampled images at different time steps more quickly, thus achieving a more efficient and cleaner infrared image generation effect. Comprehensive experiments show that the method of this invention outperforms existing methods on several typical datasets. Attached Figure Description

[0052] To make the objectives, technical solutions, and advantages of the invention clearer, the invention will now be described in further detail with reference to the accompanying drawings, wherein:

[0053] Figure 1 This is a network structure diagram of the stripe removal model (KEDM) training process.

[0054] Figure 2 This is a network structure diagram of the inference process of the stripe removal model (KEDM).

[0055] Figure 3 This is a schematic diagram of the decomposition results of DWCM.

[0056] Figure 4 This is a schematic diagram of the decomposition results of DWCM.

[0057] Figure 5 The results of different algorithms on typical test images with simulated noise are shown below: the red box corresponds to the magnified area in the image, and the green circle and arrow represent residual noise or blurred image details, respectively. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but only to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0059] The following detailed explanation illustrates the specific implementation methods:

[0060] Example:

[0061] This embodiment discloses an infrared image stripe removal method based on a knowledge embedding diffusion model.

[0062] Infrared image stripe removal methods based on knowledge embedding diffusion models include:

[0063] S1: Acquire the infrared image with striped noise to be processed;

[0064] S2: Input the striped infrared image to be processed into the trained stripe removal model (hereinafter also called KEDM), and output the corresponding striped infrared image;

[0065] like Figure 1 As shown, the stripe removal model is built based on the latent diffusion model, and the training steps are as follows:

[0066] S201: Obtain the striped noisy infrared image y and the corresponding clear infrared image (ideal infrared image) x as training samples;

[0067] S202: Perform conditional encoding on the infrared image y with striped noise to obtain the conditional feature map c; encode the clear infrared image x to obtain the initial feature vector z0 of the latent space;

[0068] S203: For the initial eigenvector z0 in the latent space, standard Gaussian noise ε is continuously added at T discrete time nodes. t ~N(0,I), thus obtaining the latent eigenvector z at time t=T. T N(·) represents a Gaussian distribution, and I represents the identity matrix;

[0069] S204: Using T trained denoising networks combined with conditional feature maps c, sequentially process the latent feature vector z starting at time t = T.t Perform back-diffusion denoising until t=0 to obtain the latent feature estimate.

[0070] S205: Estimation of latent features via decoder The image was restored to its original state, resulting in a destriped infrared image.

[0071] S206: Standard Gaussian noise ε added at each time point t The predicted noise ε output by the denoising network at the corresponding time step θ (z t The difference between z, t, and c is used to calculate the noise loss function; t This represents the intermediate latent eigenvector at time t;

[0072] S207: Based on clear infrared image x and destriated infrared image Calculate the prior loss function by introducing directional priors to fringe noise;

[0073] S208: Calculate the total loss function using the noise loss function and the prior loss function, and then optimize the parameters of the denoising network in reverse.

[0074] S209: Repeat steps S201 to S208 to iteratively train the denoising network until it converges or reaches the preset number of iterations;

[0075] like Figure 2 As shown, after training, the processing steps of the stripe removal model are as follows:

[0076] S211: Encode the infrared image with striped noise to be processed to obtain the latent feature vectors at each time point;

[0077] S212: Perform conditional encoding on the infrared image with striped noise to be processed to obtain a conditional feature map;

[0078] S213: By combining the trained T denoising networks with the conditional feature map, the latent feature vectors starting from time t=T are sequentially back-diffusion denoised until time t=0, and the latent feature estimates are obtained.

[0079] S214: The latent feature estimation is recovered through the decoder to obtain the destriped infrared image;

[0080] S3: Output the destriped infrared image as the stripe removal result of the infrared image with striped noise to be processed.

[0081] This invention creatively proposes a novel network model framework for infrared image stripe removal that combines prior knowledge with a diffusion model. First, by incorporating the spatial distribution characteristics of stripe noise into the diffusion model for constraint, and utilizing the progressive iterative sampling method of the diffusion model, it can generate better destriped output images. To our knowledge, this is the first infrared image destriping method based on a diffusion model. Second, this invention introduces a directional wavelet convolution module (DWCM) as the model's prior knowledge. DWCM can fuse stripe direction prior with semantic information, supplementing the calculation of the model's prior knowledge loss function. The directional constraint guides the generation direction of the diffusion model, forming a composite infrared destriping mode driven by both knowledge and data, thereby further improving the generalization performance of the stripe removal model (KEDM). Finally, to improve the model's training speed and reduce parameter complexity, this invention also introduces an improved latent diffusion model. By encoding and decoding the input and output images separately in a latent feature space, it can generate sampled images at different time steps more quickly, thus achieving a more efficient and cleaner infrared image generation effect. Comprehensive experiments show that the method of this invention outperforms existing methods on several typical datasets.

[0082] To better illustrate the technical solution of the present invention, this embodiment is described in the following parts.

[0083] In this embodiment, the common mathematical model for non-uniform noise in an infrared focal plane array (IRFPA) is described by the following formula:

[0084] y = x + s + n + r;

[0085] Where y is the observed infrared stripe image; x is the ideal, clear infrared image; s represents stripe noise; and n represents Gaussian noise, all of which belong to Fixed Pattern Noise (FPN). Meanwhile, r represents residual temporal noise, which is generally considered to have a small impact and can be ignored. The infrared destriping method proposed in this invention mainly aims to gradually establish the connection between the degraded infrared stripe image and the ideal infrared image (Ground Truth, GT), and to gradually refine it during the potential diffusion propagation process.

[0086] I. Encoding, Conditional Encoding, and Decoding (Recovery)

[0087] In this embodiment, encoding is performed using a VAE variational autoencoder. Here, z0 = E(x), and E(·) is the VAE variational autoencoder. The conditional encoder τ... θ Encoding the infrared image y with striped noise yields a conditional feature map c = τ. θ (y). Recovery is performed using a VAE decoder, where...

[0088] II. Forward Diffusion Process

[0089] In this embodiment, T discrete time points are sampled, and standard Gaussian noise ε is randomly generated at each time point. t Starting from time t=0, the standard Gaussian noise ε corresponding to each time point is sequentially recorded. t Layer by layer, these vectors are added to the latent space vector z0 until time t = T, generating T intermediate latent feature vectors z1 to z2. T .

[0090] During the forward diffusion process, the initial feature distribution q(z0) of the input is transformed into a series of latent space vector distributions q(z) by continuously adding Gaussian noise. T ), q(z) T The distribution of ) follows a fixed Markov chain, which can be modeled as:

[0091] The formula is expressed as:

[0092]

[0093] In the formula: N(·) represents a Gaussian distribution; I represents the identity matrix; ε T ~N(0,I),α i ∈(0,1);

[0094] By using the reparameterization method, we can characterize z from a given z0 to any time T by simplifying the intermediate steps. T The distribution of is calculated as follows:

[0095]

[0096] in

[0097] III. Reverse Diffusion Process

[0098] In this embodiment, T denoising networks are set up, each corresponding to one of the T time points; at time t = T, the first denoising network will process the intermediate latent feature vector z. T The conditional feature map c is used as input for backdiffusion denoising, and the corresponding latent feature estimate is output. Subsequent denoising networks sequentially use the latent feature estimate and conditional feature map c output by the previous denoising network as input for backdiffusion denoising until time t=0, finally obtaining the latent feature estimate.

[0099] For the reverse diffusion process, based on the characteristics of the diffusion model, a Markov chain sampling process with Gaussian noise is also formed. During this process, a denoising neural network ε is used. θ (U-net network model selected) can also distribute the latent vector p θ (z T Then switch back to the initial data estimation. The corresponding formula is:

[0100]

[0101] Where: μ θ and σ θ is the mean and variance of the Gaussian distribution at time t; c is the conditional input component, which is related to the estimated characteristics at time t. Connections are made along the channel dimension to guide the denoising network model ε. θ (z t (t,c) Correctly predict the Gaussian noise ε added at each step. t .

[0102] Characteristics at time t Corresponding correction estimation results It can be simplified as follows:

[0103]

[0104] IV. Noise Loss Function

[0105] In this embodiment, the noise loss function is expressed as follows:

[0106] L noise =||ε t -ε θ (z t ,t,c)|| 2 ;

[0107] In the formula: L noise This represents noise loss, specifically the mean squared error (MSE) loss; ε t ε represents the added Gaussian noise; θ (z t (t,c) represents the predicted noise output of the denoising network, z t Let represent the intermediate latent feature vector at time t, and c represent the conditional feature map.

[0108] The optimal parameters θ obtained from training the denoising network * Represented as:

[0109] After the denoising network model is trained, the model ε is used θThis allows us to obtain the mean μ of the Gaussian distribution at each time step. θ and σ θ Then, the initial latent feature estimate is recovered through repeated iterative calculations. Finally, by decoding with a matching VAE decoder, the striped destriped infrared image (infrared corrected image) can be restored.

[0110] V. Prior Loss Function

[0111] In this embodiment, considering the obvious directionality of the stripe noise s in the infrared image, we designed and embedded a DWCM module when using the KEDM model for denoising. This module introduces the directional prior of the stripe noise during model training, thereby guiding the model to further eliminate the stripe noise. DWCM is a multi-scale discrete Haar wavelet convolution operator, comprising four convolution filters, each defined as follows:

[0112]

[0113] In the formula: f LL It is a low-pass filter, corresponding to the low-frequency information of the input image; f LH f HL and f HH These are three high-pass filters, corresponding to the high-frequency information in the horizontal (LH), vertical (HL), and diagonal (HH) directions of the input image, respectively;

[0114] Therefore, for a given infrared image I, multi-scale wavelet decomposition results in different directions can be obtained using a convolution filter, and its expression is:

[0115]

[0116] In the formula: Conv(·) represents the convolution operation, corresponding to the above 4 discrete wavelet convolutions, which are used to further decompose the low-frequency components at the current (i-1) scale into 4 components at the i scale; These represent the output results of the infrared image at the i-th scale in the wavelet low-frequency, horizontal, vertical, and diagonal directions, respectively.

[0117] By using wavelet convolution to perform convolution calculations on infrared input images at different scales, the image can be conveniently decomposed into sub-bands of different directions. (See below.) Figure 3 As shown, by decomposing the noisy image, it can be clearly seen that the stripe noise is concentrated in... In the sub-band components, there are almost no stripes in the other three wavelet components. Given this distinct directional distribution characteristic of the stripe noise distribution, we can use it as a significant prior knowledge when training the KEDM model. By constraining the wavelet vertical component between the ground truth image and the denoising result, we can better guide the diffusion model to generate the ideal destriped result. The corresponding process is as follows: Figure 4 As shown.

[0118] Specifically, the prior loss function is calculated through the following steps:

[0119] S2071: The infrared clear image x and the destriped infrared image x are processed using a Directional wavelet convolution module (DWCM). Perform wavelet decomposition at multiple scales to obtain multi-scale wavelet decomposition results in different directions;

[0120] S2072: Extract the sharp infrared image x and the destriped infrared image. Wavelet vertical components x at various scales HL and

[0121] S2073: Using wavelet vertical component x HL and (To minimize the prior loss of both during model training) Calculate the prior loss function;

[0122] The formula is expressed as:

[0123]

[0124] In the formula: L prior This is the prior loss. The purpose of setting the prior loss is to make the diffusion model consciously focus on the vertical component of the denoised image during training, so that it approximates the reference stripe-free image as closely as possible, thereby achieving the stripe removal effect of the model in back diffusion.

[0125] VI. Total Loss Function

[0126] In this embodiment, the formula for calculating the total loss function is as follows:

[0127] L total =L noise +λL prior ;

[0128] In the formula: L total λ represents the total loss function; λ is an adjustable hyperparameter, whose value can generally be set to 1.

[0129] VII. Experimental Instructions

[0130] To better illustrate the advantages of the technical solution of the present invention, the following experiment is disclosed in this embodiment.

[0131] 1. Dataset and Experiment Setup

[0132] To cover a wider range of scenarios, this experiment first selected 680 infrared images from three datasets to form a dataset: CVC (from “Y. Socarr′as, S. Ramos, DV′azquez, AML′opez, and T. Gevers, “Adapting pedestrian detection from synthetic to far infrared images,” in ICCV Workshops, vol. 3, 2013.”), INFRARED (from “Z. He, Y. Cao, Y. Dong, J. Yang, Y. Cao, and CLTisse, “Single-image-based nonuniformity correction of uncooled long-wave infrared detectors: a deep-learning approach,” Appl Opt, vol. 57, no. 18, pp. D155-D164, Jun 20, 2018.”), and FLIR (from “F. Teledyne, Free teledyne FLIR thermal dataset for algorithm training, Teledyne FLIR (2018).”). Simultaneously, using data augmentation strategies including rotation, scaling, cropping, and contrast adjustment, the original dataset was uniformly cropped to a 256×256 image size, generating 130,268 image patches. In this way, we can randomly select 70% of the image patches for training and the remaining 30% for testing.

[0133] We implemented the proposed KEDM model using the PyTorch 1.10.0 framework and conducted experiments on two Nvidia GeForce3090 GPUs, one Intel Core i9-10850K CPU, and 64GB of RAM. For the basic diffusion model architecture, the pre-trained VAE encoder encodes both the ground truth (GT) image and the non-uniformly degraded image into latent space vectors of size 64×64, and self-attention layers are set at spatial resolutions of 8, 16, and 32 in the Denoising-UNet. Consistent with the common training settings for diffusion models, we also set 1000 forward diffusion time steps for training and 200 backward diffusion time steps for inference, and optimized using the Adam optimizer. During this process, the batch size, learning rate, and number of iterations were set to 48, 1e-5, and 1,000,000, respectively.

[0134] 2. Comparison with other typical methods

[0135] To verify the performance of the proposed algorithm, we designed KEDM and compared it with several state-of-the-art infrared image destriping methods (from “Y. Cao, MY Yang, and C.-L. Tisse, “Effective strip noise removal for low-textured infrared images based on 1-d guided filtering,” IEEE Trans. Circuits Syst. Video Technol., vol. 26, no. 12, pp. 2176–2188, 2015.”, “L. Liu, L. Xu, and H. Fang, “Simultaneous intensity bias estimation and stripe noise removal in infrared images using the global and local sparsity constraints,” IEEE Trans. Geosci. Remote Sens., vol. 58, no. 3, pp. 1777–1789, Mar. 2020.”, “Z. He, Y. Cao, Y. Dong, J. Yang, Y. Cao, and C. L. Tisse, “Single-image-based nonuniformity correction of…”). uncooled long-wave infrared detectors: a deep-learning approach," Appl Opt, vol.57, no.18, pp.D155-D164, Jun 20, 2018.", "Y.Chang, L.Yan, L.Liu, H.Fang, and S.Zhong, "Infrared Aerothermal Nonuniform Correction via DeepMultiscale Residual Network," IEEE Geoscience and Remote Sensing Letters, vol.16, no.7, pp.1120-1124, 2019.", "J.Song,J.-H.Jeong,D.-S.Park,H.-H.Kim,D.-C.Seo,and JCA comparison was made using Ye's work, “Unsupervised Denoising for Satellite Imagery Using Wavelet Directional CycleGAN,” IEEE Transactions on Geoscience and Remote Sensing, vol. 59, no. 8, pp. 6823-6839, 2021. We used the default parameters for each method to ensure a fair comparison. For quantitative metrics, to maintain consistency with related work, we used peak signal-to-noise ratio (PSNR) and structural similarity index measurement (SSIM) to evaluate model performance. Higher PSNR and SSIM scores indicate better destriating of infrared images.

[0136] Table 1 below lists the quantitative comparison results of various methods. During the testing process, we also used a cubic hybrid degradation model (from "C. Wang, M. Xu, Y. Jiang, G. Zhang, H. Cui, L. Li, and D. Li, "Translution-SNet: A Semi-supervised Hyperspectral Image Stripe Noise Removal Based on Transformer and CNN," IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1-14, 2022.) and "S. Yang, H. Qin, S. Yuan, X. Yan, and H. Japa Rahmani, "DestripeCycleGAN: Stripe Simulation CycleGAN for Unsupervised Infrared Image Destriping," 2024.) to simulate Gaussian noise and uniform stripe noise in infrared images, respectively. For Gaussian noise, the standard deviation σ was set to 0.05, and for uniform noise, the mean μ was set to 0.1. The best results in Table 1 are shown in bold. We can clearly see that in most cases, the KEDM destriping results proposed in this invention are superior to those of other algorithms.

[0137] Table 1. PSNR / SSIM values ​​of different methods under two stripe noise conditions on three test sets.

[0138]

[0139] As shown in Table 1, the PSNR and SSIM indices of the KEDM proposed in this invention are improved. Compared with GF, GLSC, DLS, TSWEU, and WCGAN, the average PSNR index of KEDM increased by 12.9% (from 35.52 to 40.10), 10.7% (from 36.21 to 40.10), 13.1% (from 35.46 to 40.10), 6.6% (from 37.60 to 40.10), and 2.1% (from 39.27 to 40.10), respectively. For the average SSIM index, KEDM increased by 3.5% (from 0.9558 to 0.9892), 1.9% (from 0.9711 to 0.9892), 2.8% (from 0.9621 to 0.9892), 1.1% (from 0.9787 to 0.9892), and 0.6% (from 0.9837 to 0.9892), respectively. Although the increase in the SSIM index by KEDM was limited, its average result reached around 0.99, and its performance was the best on all three datasets. This demonstrates that the proposed method does indeed have a better destriping effect on different infrared images.

[0140] Figure 5 Several representative images from CVC, INFRARED, and FLIR are shown, along with their destriating effects under different algorithms. In sub-figures (c)-(g), we can observe that some algorithms, such as GF, DLS, and TSWEU, exhibit some residual striating noise in the areas corresponding to the green circles, which limits the image's signal-to-noise ratio. For the results of GLSC and WCGAN, although the residual noise is relatively small, as seen from the green arrows in sub-figures (d) and (g), their destriating results blur local areas of the image, and some detailed textures in the scene are lost due to excessive smoothing, even resulting in artifacts. Our KEDM method, however, is the only one that effectively eliminates striating noise while preserving detailed edges. As shown in sub-figure (h), its processing result almost completely eliminates striating noise, and the scene details in the image are basically consistent with the GT image. For example, the railings along the street and the Z-shaped textures on the window frames are well preserved, thus demonstrating the superior performance of our proposed method.

[0141] 3. Ablation test

[0142] To further demonstrate the effectiveness of each module of the KEDM algorithm proposed in this invention, we conducted an ablation experiment using the CVC dataset with uniformly distributed stripe noise of μ = 0.1. The corresponding results are shown in Table 2 below. Compared with the original diffusion model (DM), we mainly introduced LDM and DWCM modules into the network structure. Therefore, we first trained the original diffusion model using the same initialization parameters without using the DWCM module. The results are shown in the first row of Table 2. It can be seen that although the original diffusion model also performs well in terms of destriating performance in the spatial domain, it has a very large computational load due to directly performing diffusion processing on the original size infrared image, resulting in the slowest inference speed. When using LDM instead of the original DM, the various indicators are shown in the second row, with PSNR and SSIM indicators being similar to DM, but the inference time is significantly reduced to about 1 / 30 of the DM model. This shows that replacing DM with LDM can indeed greatly improve performance while ensuring destriating performance. Model computation speed: When we introduce the DWCM module into DM, as shown in the third row, the corresponding PSNR and SSIM indices are further improved. This shows that introducing the DWCM module as a prior constraint can indeed better guide the directional diffusion direction of the model and obtain better destriating effect. Finally, when we combine LDM and DWCM module, that is, when using the entire KEDM model, the corresponding results are shown in the fourth row. It can be seen that its PSNR and SSIM indices have reached the optimal results, and the time consumed is only 1.02s, which is very close to the fastest 0.97s. This also shows that the method of the present invention can achieve a good balance between destriating effect and computational efficiency.

[0143] Table 1. Comparison of ablation experimental results under different settings (bold indicates best results)

[0144]

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit the technical solutions. Those skilled in the art should understand that any modifications or equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention should be covered within the scope of the claims of the present invention.

Claims

1. A method for removing stripes from infrared images based on a knowledge embedding diffusion model, characterized in that, include: S1: Acquire the infrared image with striped noise to be processed; S2: Input the striped infrared image with noise to be processed into the trained stripe removal model, and output the corresponding striped infrared image; The stripe removal model is built based on the latent diffusion model, and the training steps are as follows: S201: Obtain the striped, noisy infrared image y and the corresponding clear infrared image x as training samples; S202: Perform conditional encoding on the infrared image y with striped noise to obtain the conditional feature map c; encode the clear infrared image x to obtain the initial feature vector z0 of the latent space; S203: For the initial eigenvector z0 in the latent space, standard Gaussian noise ε is continuously added at T discrete time nodes. t ~N(0,I), thus obtaining the latent eigenvector z at time t=T. T N(·) represents a Gaussian distribution, and I represents the identity matrix; S204: Using T trained denoising networks combined with conditional feature maps c, sequentially process the latent feature vector z starting at time t = T. t Perform back-diffusion denoising until t=0 to obtain the latent feature estimate. S205: Estimation of latent features via decoder The image was restored to its original state, resulting in a destriped infrared image. S206: Standard Gaussian noise ε added at each time point t The predicted noise ε output by the denoising network at the corresponding time step θ (z t The difference between z, t, and c is used to calculate the noise loss function; t This represents the intermediate latent eigenvector at time t; S207: Based on clear infrared image x and destriated infrared image Calculate the prior loss function by introducing directional priors to fringe noise; In step S207, the prior loss function is calculated through the following steps: S2071: The infrared clear image x and the destriped infrared image x are processed using a directional wavelet convolution module, respectively. Perform wavelet decomposition at multiple scales to obtain multi-scale wavelet decomposition results in different directions; S2072: Extract the sharp infrared image x and the destriped infrared image. Wavelet vertical components x at various scales HL and S2073: Using wavelet vertical component x HL and Calculate the prior loss function; The formula is expressed as: In the formula: L prior For prior loss; In S2071, the directional wavelet convolution module includes four convolutional filters, which are defined as follows: In the formula: f LL It is a low-pass filter, corresponding to the low-frequency information of the input image; f LH f HL and f HH These are three high-pass filters, corresponding to the high-frequency information in the horizontal, vertical, and diagonal directions of the input image, respectively; For a given infrared image I, the multi-scale wavelet decomposition results in different directions are obtained using a convolution filter, and their expression is: In the formula: Conv(·) represents the convolution operation, corresponding to 4 discrete wavelet convolutions, which are used to further decompose the low-frequency components at the current (i-1) scale into 4 components at the i scale; These represent the output results of the infrared image at the i-th scale in the wavelet low-frequency, horizontal, vertical, and diagonal directions, respectively. S208: Calculate the total loss function using the noise loss function and the prior loss function, and then optimize the parameters of the denoising network in reverse. S209: Repeat steps S201 to S208 to iteratively train the denoising network until it converges or reaches the preset number of iterations; After training, the stripe removal model undergoes the following processing steps: S211: Encode the infrared image with striped noise to be processed to obtain the latent feature vectors at each time point; S212: Perform conditional encoding on the infrared image with striped noise to be processed to obtain a conditional feature map; S213: By combining the trained T denoising networks with the conditional feature map, the latent feature vectors starting from time t=T are sequentially back-diffusion denoised until time t=0, and the latent feature estimates are obtained. S214: The latent feature estimation is recovered through the decoder to obtain the destriped infrared image; S3: Output the destriped infrared image as the stripe removal result of the infrared image with striped noise to be processed.

2. The infrared image stripe removal method based on the knowledge embedding diffusion model as described in claim 1, characterized in that: In steps S202 and S211, encoding is performed using a VAE variational autoencoder.

3. The infrared image stripe removal method based on the knowledge embedding diffusion model as described in claim 1, characterized in that: In steps S202 and S212, the infrared image with striped noise is conditionally encoded using a conditional encoder.

4. The infrared image stripe removal method based on the knowledge embedding diffusion model as described in claim 1, characterized in that: In step S203, T discrete time points are sampled, and standard Gaussian noise ε is randomly generated at each time point. t Starting from time t=0, the standard Gaussian noise ε corresponding to each time point is sequentially recorded. t Layer by layer, these vectors are added to the latent space vector z0 until time t = T, generating T intermediate latent feature vectors z1 to z2. T .

5. The infrared image stripe removal method based on the knowledge embedding diffusion model as described in claim 1, characterized in that: In step S204, T denoising networks are set up, each corresponding to one of the T time points; at time t=T, the first denoising network decomposes the intermediate latent feature vector z. T The conditional feature map c is used as input for backdiffusion denoising, and the corresponding latent feature estimate is output. Subsequent denoising networks sequentially use the latent feature estimate and conditional feature map c output by the previous denoising network as input for backdiffusion denoising until time t=0, finally obtaining the latent feature estimate.

6. The infrared image stripe removal method based on the knowledge embedding diffusion model as described in claim 1, characterized in that: In steps S205 and S214, recovery is performed using the VAE decoder.

7. The infrared image stripe removal method based on the knowledge embedding diffusion model as described in claim 1, characterized in that: In step S206, the formula for the noise loss function is expressed as follows: L noise =||ε t -ε θ (z t ,t,c)|| 2 ; In the formula: L noise Indicates noise loss; ε t ε represents the added Gaussian noise; θ (z t (t,c) represents the predicted noise output of the denoising network, z t Let represent the intermediate latent feature vector at time t, and c represent the conditional feature map.

8. The infrared image stripe removal method based on the knowledge embedding diffusion model as described in claim 1, characterized in that: In step S208, the formula for calculating the total loss function is as follows: THE total =L noise +λL prior ; In the formula: L total Let L represent the total loss function; λ is an adjustable hyperparameter; L prior For prior loss; L noise This indicates noise loss.

Images