Infrared image stripe removing method based on knowledge embedding diffusion model

Abstract

The invention discloses an infrared image stripe removal method based on a knowledge embedding diffusion model. The method comprises the following steps: S1, acquiring a to-be-processed stripe noise infrared image; s2, inputting the infrared image with the stripe noise into the trained stripe removal model, and outputting a corresponding stripe-removed infrared image; when the stripe removal model is trained, parameters of a denoising network are jointly optimized based on a noise loss function and a prior loss function introducing stripe noise direction prior; and S3, outputting the output stripe-removed infrared image as a stripe removal result of the to-be-processed stripe noise infrared image. On the basis of the diffusion model, a directional wavelet convolution module is introduced to serve as knowledge priori of the model, the module can fuse stripe direction priori and semantic information, a knowledge priori loss function of the model is complemented and calculated, the generation direction of the diffusion model is directionally constrained and guided, and the generation efficiency of the diffusion model is improved. And a knowledge and data dual-driven composite infrared stripe removing mode is formed.

Description

Technical Field

[0001] The present invention relates to the field of infrared imaging and artificial intelligence big data technology, and in particular to a method for removing infrared image stripes based on a knowledge embedding diffusion model. Background Art

[0002] Infrared imaging (IR) technology has important applications in military guidance, aerospace, remote sensing detection, security monitoring and other fields due to its outstanding advantages such as all-weather operation, high concealment and strong anti-interference ability. However, due to the limitations of the manufacturing level and processing accuracy of the infrared focal plane array (IRFPA), the actual response of each detection unit in the infrared imaging system is often difficult to be completely consistent, resulting in the output image of the infrared imaging system often having many non-uniform stripe noises even under uniform radiation, which is called the fixed pattern noise (FPN) of the infrared focal plane array. The existence of FPN will seriously affect the visual effect of the output image, greatly reduce the temperature resolution and detection response capability of the infrared imaging system, and ultimately restrict the further application of infrared imaging technology. Therefore, it is necessary to correct the non-uniform stripe noise of the acquired infrared detection image in order to better perform the subsequent image target recognition task.

[0003] At present, the research on algorithms for removing streaks from infrared images can be roughly divided into two categories: 1) methods based on traditional model optimization; 2) methods based on deep learning. For various methods in 1), such as filtering methods, minimizing energy constraints methods, and response statistics methods, the core is mainly to use a single-frame or multi-frame iterative optimization algorithm based on the visual difference between the image scene and the streak noise, and to separate the streak noise and the image scene through a manually designed feature extraction operator. Although this type of algorithm is simple in design, it overly relies on artificially designed rough features to describe image attributes, so it has poor adaptability and is difficult to achieve acceptable correction effects for streak noises of different intensities and types at the same time. For various methods in 2), the core is to use the powerful feature representation ability of convolutional neural networks (CNNs) and sample learning to mine the spatiotemporal distribution characteristics of streak noise from a large amount of sample data, thereby eliminating it. In recent years, some methods have also been combined with deep generation models based on GAN networks to achieve the effect of self-supervision or unsupervised learning. Although various streak removal methods based on deep learning have shown powerful feature learning and modeling capabilities, they have also had some insurmountable defects so far. For example, the CNN-based method, due to the limited receptive field size of the general convolution kernel, restricts its prediction accuracy for complex semantic information, resulting in the model being unable to completely distinguish between stripe noise and background texture; and various generative models based on GAN will face challenges such as mode collapse, model optimization difficulties and gradient disappearance, which will have an adverse effect on their task of removing stripes from infrared images.

[0004] In recent years, a new architecture called the diffusion model in deep generative models has shown great application potential in various computer vision tasks. The applicant found that compared with the GAN model, the diffusion model can generate more realistic detail textures and output better image quality in various low-level visual tasks such as image denoising, restoration, and super-resolution, and will not encounter thorny problems such as unstable training and difficult model optimization like the GAN model. The diffusion model adopts a progressive refinement feature learning strategy, which can simulate the feature distribution of the target image domain through iterative sampling, which is actually consistent with the physical process of infrared image stripe removal. Therefore, how to design a method for infrared image stripe removal based on the diffusion model is a technical problem that needs to be solved urgently. Summary of the invention

[0005] In view of the deficiencies of the above-mentioned prior art, the technical problem to be solved by the present invention is: how to provide a method for removing stripes from infrared images based on a knowledge embedded diffusion model. On the basis of the diffusion model, a directional wavelet convolution module is introduced as the knowledge prior of the model. The module can fuse the stripe direction prior with the semantic information, supplement the knowledge prior loss function of the calculation model, and guide the generation direction of the diffusion model with directional constraints, thereby forming a composite infrared stripe removal mode driven by both knowledge and data, thereby further improving the generalization performance of the model.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] A method for removing infrared image stripes based on knowledge embedding diffusion model, comprising:

[0008] S1: Acquire the infrared image with stripe noise to be processed;

[0009] S2: Input the infrared image with stripe noise to be processed into the trained stripe removal model, and output the corresponding stripe-free infrared image;

[0010] The stripe removal model is built based on the potential diffusion model, and the training steps are as follows:

[0011] S201: Acquire an infrared image y with stripe noise and a corresponding infrared clear image x as training samples;

[0012] S202: Conditionally encode the infrared image y with stripe noise to obtain the conditional feature map c; encode the infrared clear image x to obtain the initial feature vector z of the latent space 0 ;

[0013] S203: Sequentially calculate the initial feature vector z of the latent space 0 Continuously add standard Gaussian noise ε at T discrete time nodes t ~N(0,I), and obtain the potential feature vector z at time t=T T ; N(·) represents Gaussian distribution, I represents the identity matrix;

[0014] S204: Using T trained denoising networks combined with conditional feature graphs c, the potential feature vector z starting at time t = T is sequentially denoised. t Perform reverse diffusion denoising until time t = 0 to obtain the potential feature estimate

[0015] S205: Estimation of latent features by decoder Restore and get the stripe-free infrared image

[0016] S206: Standard Gaussian noise ε added at each time node t And the predicted noise ε output at the corresponding time of the denoising network θ (z t ,t,c) difference calculation noise loss function; z t represents the intermediate latent feature vector at time t;

[0017] S207: Based on infrared clear image x and de-streak infrared image The direction prior of the stripe noise is introduced to calculate the prior loss function;

[0018] S208: Calculate the total loss function through the noise loss function and the prior loss function and reversely optimize the parameters of the denoising network;

[0019] S209: repeating steps S201 to S208 to iteratively train the denoising network until convergence or reaching a preset number of iterations;

[0020] After training is complete, the processing steps of the streak removal model are as follows:

[0021] S211: Encode the infrared image with stripe noise to be processed to obtain the potential feature vector of each time node;

[0022] S212: Conditionally encode the infrared image with stripe noise to be processed to obtain a conditional feature map;

[0023] S213: by combining the trained T denoising networks with the conditional feature graph, the potential feature vectors starting from time t=T are subjected to reverse diffusion denoising until time t=0 to obtain a potential feature estimate;

[0024] S214: restoring the potential feature estimation through the decoder to obtain a stripe-free infrared image;

[0025] S3: Outputting the outputted de-striation infrared image as the stripe removal result of the infrared image with stripe noise to be processed.

[0026] Preferably, in step S202 and step S211, encoding is performed using a VAE variational autoencoder.

[0027] Preferably, in step S202 and step S212, the infrared image with striped noise is conditionally encoded by a conditional encoder.

[0028] Preferably, in step S203, T discrete time nodes are sampled, and standard Gaussian noise ε is randomly generated at each time node. t ; Starting from time t = 0, the standard Gaussian noise ε corresponding to each moment is t Add to the latent space vector z layer by layer0 Until time t = T, T intermediate potential feature vectors z are generated 1 to z T .

[0029] Preferably, in step S204, T denoising networks corresponding to the T time nodes are set; at time t=T, the first denoising network converts the intermediate potential feature vector z T and conditional feature map c as input for reverse diffusion denoising, and output the corresponding potential feature estimate The subsequent denoising network takes the potential feature estimate and conditional feature map c output by the previous denoising network as input for reverse diffusion denoising until time t = 0, and finally obtains the potential feature estimate

[0030] Preferably, in step S205 and step S214, recovery is performed through a VAE decoder.

[0031] Preferably, in step S206, the noise loss function is expressed as:

[0032] L noise =||ε t -ε θ (z t ,t,c)|| 2 ;

[0033] Where: L noise represents noise loss; ε t represents the added Gaussian noise; ε θ (z t ,t,c) represents the predicted noise output of the denoising network, z t represents the intermediate potential feature vector at time t, and c represents the conditional feature map.

[0034] Preferably, in step S207, the prior loss function is calculated by the following steps:

[0035] S2071: Directed wavelet convolution module is used to process the infrared clear image x and the de-striated infrared image Perform wavelet decomposition at multiple scales to obtain multi-scale wavelet decomposition results in different directions;

[0036] S2072: Extract infrared clear image x and de-streak infrared image The vertical component x of the wavelet at each scale HL and

[0037] S2073: Through the wavelet vertical component x HL and Calculate the prior loss function;

[0038] The formula is:

[0039]

[0040] Where: L prior is the prior loss.

[0041] Preferably, in S2071, the directional wavelet convolution module includes four convolution filters, which are defined as follows:

[0042]

[0043] Where: f LL is a low-pass filter corresponding to the low-frequency information of the input image; f LH 、f HL and f HH There are three high-pass filters, corresponding to the high-frequency information in the horizontal, vertical and diagonal directions of the input image;

[0044] For a given infrared image I, the multi-scale wavelet decomposition results in different directions are obtained by using convolution filters, and the expression is:

[0045]

[0046] Where: Conv(·) represents the convolution operation, corresponding to 4 discrete wavelet convolutions, which is used to further decompose the low-frequency component of the current (i-1) scale into 4 components at the i scale; They are the output results of the infrared image at the i-th scale in the wavelet low frequency, horizontal, vertical and diagonal directions respectively.

[0047] Preferably, in step S208, the calculation formula of the total loss function is as follows:

[0048] L total =L noise +λL prior ;

[0049] Where: L total represents the total loss function; λ is an adjustable hyperparameter; L prior is the prior loss; L noise Indicates noise loss.

[0050] Compared with the prior art, the infrared image stripe removal method based on the knowledge embedding diffusion model in the present invention has the following beneficial effects:

[0051] The present invention creatively proposes a new network model framework for infrared image stripe removal that combines knowledge prior and diffusion model. First, the present invention introduces the spatial distribution characteristics of stripe noise into the diffusion model for constraint, and utilizes the progressive iterative sampling method of the diffusion model to better generate the stripe-removed output image result. As far as we know, this is the first infrared image stripe removal method based on the diffusion model. Secondly, the present invention introduces a directional wavelet convolution module (DWCM) as the knowledge prior of the model. The DWCM can fuse the stripe direction prior with the semantic information, supplement the knowledge prior loss function of the calculation model, and guide the generation direction of the diffusion model with directional constraints, forming a composite infrared stripe removal mode driven by both knowledge and data, thereby further improving the generalization performance of the stripe removal model (KEDM). Finally, in order to improve the training speed of the model and reduce the parameter complexity of the model, the present invention also introduces an improved latent diffusion model. By encoding and decoding the input and output images in a latent feature space respectively, the sampled images of different time steps are generated faster, thereby achieving a more efficient clean infrared image generation effect. Comprehensive experiments show that the method of the present invention is superior to the existing methods on several typical data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to make the purpose, technical solution and advantages of the invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0053] Figure 1 The network structure diagram of the stripe removal model (KEDM) training process.

[0054] Figure 2 Figure 2 is the network structure diagram of the KEDM reasoning process.

[0055] Figure 3 Schematic diagram of the decomposition results of DWCM.

[0056] Figure 4 Schematic diagram of the decomposition results of DWCM.

[0057] Figure 5 The stripe removal results of different algorithms on a typical test image with simulated noise: the red box corresponds to the locally enlarged area in the image, and the green circle and arrow represent the residual noise or blurred image details, respectively. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but only represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0059] The following is a further detailed description through specific implementation methods:

[0060] Example:

[0061] This embodiment discloses a method for removing infrared image stripes based on a knowledge embedding diffusion model.

[0062] The infrared image stripe removal method based on knowledge embedding diffusion model includes:

[0063] S1: Acquire the infrared image with stripe noise to be processed;

[0064] S2: Input the infrared image with streak noise to be processed into the trained streak removal model (hereinafter also referred to as KEDM), and output the corresponding streak-free infrared image;

[0065] like Figure 1 As shown in the figure, the stripe removal model is built based on the potential diffusion model, and the training steps are as follows:

[0066] S201: Acquire an infrared image y with stripe noise and a corresponding infrared clear image (ideal infrared image) x as a training sample;

[0067] S202: Conditionally encode the infrared image y with stripe noise to obtain the conditional feature map c; encode the infrared clear image x to obtain the initial feature vector z of the latent space 0 ;

[0068] S203: Sequentially calculate the initial feature vector z of the latent space 0 Continuously add standard Gaussian noise ε at T discrete time nodes t ~N(0,I), and obtain the potential feature vector z at time t=T T ; N(·) represents Gaussian distribution, I represents the identity matrix;

[0069] S204: Using T trained denoising networks combined with conditional feature graphs c, the potential feature vector z starting at time t = T is sequentially denoised. t Perform reverse diffusion denoising until time t = 0 to obtain the potential feature estimate

[0070] S205: Estimation of latent features by decoder Restore and get the stripe-free infrared image

[0071] S206: Standard Gaussian noise ε added at each time node t And the predicted noise ε output at the corresponding time of the denoising network θ (z t ,t,c) difference calculation noise loss function; z t represents the intermediate latent feature vector at time t;

[0072] S207: Based on infrared clear image x and de-streak infrared image The direction prior of the stripe noise is introduced to calculate the prior loss function;

[0073] S208: Calculate the total loss function through the noise loss function and the prior loss function and reversely optimize the parameters of the denoising network;

[0074] S209: repeating steps S201 to S208 to iteratively train the denoising network until convergence or reaching a preset number of iterations;

[0075] like Figure 2 As shown in the figure, after the training is completed, the processing steps of the stripe removal model are as follows:

[0076] S211: Encode the infrared image with stripe noise to be processed to obtain the potential feature vector of each time node;

[0077] S212: Conditionally encode the infrared image with stripe noise to be processed to obtain a conditional feature map;

[0078] S213: by combining the trained T denoising networks with the conditional feature graph, the potential feature vectors starting from time t=T are subjected to reverse diffusion denoising until time t=0 to obtain a potential feature estimate;

[0079] S214: restoring the potential feature estimation through the decoder to obtain a stripe-free infrared image;

[0080] S3: Outputting the outputted de-striation infrared image as the stripe removal result of the infrared image with stripe noise to be processed.

[0081] The present invention creatively proposes a new network model framework for infrared image stripe removal that combines knowledge prior and diffusion model. First, the present invention introduces the spatial distribution characteristics of stripe noise into the diffusion model for constraint, and utilizes the progressive iterative sampling method of the diffusion model to better generate the stripe-removed output image result. As far as we know, this is the first infrared image stripe removal method based on the diffusion model. Secondly, the present invention introduces a directional wavelet convolution module (DWCM) as the knowledge prior of the model. The DWCM can fuse the stripe direction prior with the semantic information, supplement the knowledge prior loss function of the calculation model, and guide the generation direction of the diffusion model with directional constraints, forming a composite infrared stripe removal mode driven by both knowledge and data, thereby further improving the generalization performance of the stripe removal model (KEDM). Finally, in order to improve the training speed of the model and reduce the parameter complexity of the model, the present invention also introduces an improved latent diffusion model. By encoding and decoding the input and output images in a latent feature space respectively, the sampled images of different time steps are generated faster, thereby achieving a more efficient clean infrared image generation effect. Comprehensive experiments show that the method of the present invention is superior to the existing methods on several typical data sets.

[0082] In order to better introduce the technical solution of the present invention, this embodiment is described through the following parts.

[0083] In this embodiment, a common mathematical model of non-uniform noise of an infrared focal plane array (IRFPA) is described by the following formula:

[0084] y=x+s+n+r;

[0085] Where y is the observed infrared streak image; x is the ideal clear infrared image; s represents the streak noise; n represents the Gaussian noise, all of which belong to fixed pattern noise FPN noise (Fixed Pattern Noise: FPN). At the same time, r represents the residual time domain noise, which is generally considered to have a small impact and can be ignored. The infrared streak removal method proposed in the present invention is mainly to gradually establish the connection between the infrared streak degraded image and the ideal infrared image (Ground Truth, GT), and gradually refine it in the process of potential diffusion transfer.

[0086] 1. Encoding, conditional encoding and decoding (recovery)

[0087] In this embodiment, encoding is performed by a VAE variational autoencoder. 0 =E(x), E(·) is the VAE variational self-encoder. Through the conditional encoder τ θ Encode the infrared image y with striped noise and obtain the conditional feature map c = τ θ(y). Recovered by VAE decoder, where

[0088] 2. Forward Diffusion Process

[0089] In this embodiment, T discrete time nodes are sampled and standard Gaussian noise ε is randomly generated at each time node. t ; Starting from time t = 0, the standard Gaussian noise ε corresponding to each moment is t Add to the latent space vector z layer by layer 0 Until time t = T, T intermediate potential feature vectors z are generated 1 to z T .

[0090] In the forward diffusion process, the initial feature distribution q(z 0 ) is transformed into a series of latent space vector distributions q(z T ), q(z T ) follows a stationary Markov chain and can be modeled as:

[0091] The formula is:

[0092]

[0093] Where: N(·) represents Gaussian distribution; I represents the unit matrix; ε T ~N(0,I),α i ∈(0,1);

[0094] By reparameterization, we can characterize the process from a given z to 0 At any time T z T The distribution form of is calculated as follows:

[0095]

[0096] in

[0097] 3. Reverse Diffusion Process

[0098] In this embodiment, T denoising networks corresponding to T time nodes are set; at time t=T, the first denoising network converts the intermediate potential feature vector z T and conditional feature map c as input for reverse diffusion denoising, and output the corresponding potential feature estimate The subsequent denoising network takes the potential feature estimate and conditional feature map c output by the previous denoising network as input for reverse diffusion denoising until time t = 0, and finally obtains the potential feature estimate

[0099] For the reverse diffusion process, according to the characteristics of the diffusion model, a Markov chain sampling process with Gaussian noise is also formed. In this process, a denoising neural network ε θ (U-net network model is selected) and the potential vector distribution p θ (z T ) and converted back to the original data estimate The corresponding formula is:

[0100]

[0101] Where: μ θ and σ θ is the mean and variance of the Gaussian distribution at time t; c is the conditional input component, which is related to the estimated feature at time t Connect in the channel dimension to guide the denoising network model ε θ (z t ,t,c) Correctly predict the Gaussian noise ε added at each step t .

[0102] Characteristics at time t The corresponding correction estimation results It can be simplified as follows:

[0103]

[0104] 4. Noise Loss Function

[0105] In this embodiment, the noise loss function is expressed as:

[0106] L noise =||ε t -ε θ (z t ,t,c)|| 2 ;

[0107] Where: L noise represents the noise loss, which is the mean square error (MSE) loss; ε t represents the added Gaussian noise; ε θ (z t ,t,c) represents the predicted noise output of the denoising network, z t represents the intermediate potential feature vector at time t, and c represents the conditional feature map.

[0108] The optimal parameter θ obtained by denoising network training * It is expressed as:

[0109] After the denoising network model training is completed, the model εθ You can get the mean μ of the Gaussian distribution at each time step θ and σ θ ; Then, the initial potential feature estimate is restored through repeated iterative calculations Finally, the de-striped infrared image (infrared corrected image) can be restored by decoding it through the matched VAE decoder D.

[0110] 5. Prior Loss Function

[0111] In this embodiment, considering that the stripe noise s of the infrared image has obvious directionality, we designed and embedded a DWCM module when using the KEDM model for denoising, which is used to introduce the direction prior of the stripe noise in the model training, thereby guiding the model to further eliminate the stripe noise. DWCM is a multi-scale discrete Harr wavelet convolution operator, including 4 convolution filters, each of which is defined as follows:

[0112]

[0113] Where: f LL is a low-pass filter corresponding to the low-frequency information of the input image; f LH 、f HL and f HH There are three high-pass filters, corresponding to the high-frequency information in the horizontal (LH), vertical (HL) and diagonal (HH) directions of the input image;

[0114] Therefore, for a given infrared image I, the multi-scale wavelet decomposition results in different directions can be obtained by using the convolution filter, and its expression is:

[0115]

[0116] Where: Conv(·) represents the convolution operation, corresponding to the above four discrete wavelet convolutions, which is used to further decompose the low-frequency component of the current (i-1) scale into four components at the i scale; They are the output results of the infrared image at the i-th scale in the wavelet low frequency, horizontal, vertical and diagonal directions respectively.

[0117] By using the above wavelet convolution to perform convolution calculations on infrared input images at different scales, the image can be easily decomposed into sub-bands in different directions. Figure 3 As shown in Figure 2, the noisy image is decomposed and it can be clearly seen that the stripe noise is concentrated in In the subband component of , there are almost no stripes in the other three wavelet components. Given the obvious directional distribution characteristics of stripe noise distribution, we can use it as a significant prior knowledge when training the KEDM model. By constraining the vertical component of the wavelet between the GT image and the denoising result, we can better guide the diffusion model to generate an ideal stripe removal result. The corresponding process is as follows Figure 4 shown.

[0118] Specifically, the prior loss function is calculated through the following steps:

[0119] S2071: The infrared clear image x and the de-striated infrared image x are respectively processed by the directional wavelet convolution module (DWCM). Perform wavelet decomposition at multiple scales to obtain multi-scale wavelet decomposition results in different directions;

[0120] S2072: Extract infrared clear image x and de-streak infrared image The vertical component x of the wavelet at each scale HL and

[0121] S2073: Through the wavelet vertical component x HL and (Make the prior loss of both in model training as small as possible) Calculate the prior loss function;

[0122] The formula is:

[0123]

[0124] Where: L prior The purpose of setting the prior loss is to make the diffusion model consciously focus on the vertical component of the denoised image during training, so that it can be as close to the reference stripe-free image as possible, thereby achieving the stripe removal effect of the model in reverse diffusion.

[0125] 6. Total loss function

[0126] In this embodiment, the calculation formula of the total loss function is as follows:

[0127] L total =L noise +λL prior ;

[0128] Where: L total Represents the total loss function; λ is an adjustable hyperparameter, whose value can generally be set to 1.

[0129] VII. Experimental Description

[0130] In order to better illustrate the advantages of the technical solution of the present invention, this embodiment discloses the following experiment.

[0131] 1. Dataset and Experimental Setup

[0132] In order to cover a wider range of scenarios, this experiment first selected 680 infrared images from three datasets: CVC (from “Y.Socarr′as, S.Ramos, DV′azquez, AML′opez, and T.Gevers, “Adapting pedestrian detection from syntheticto far infrared images,” in ICCV Workshops, vol.3, 2013.”), INFRARED (from “Z.He, Y.Cao, Y.Dong, J.Yang, Y.Cao, and CLTisse, “Single-image-based nonuniformity correction of uncooled long-wave infrared detectors: a deep-learningapproach,” Appl Opt, vol.57, no.18, pp.D155-D164, Jun 20, 2018.”) and FLIR (from “F.Teledyne, Free teledyne FLIR thermal dataset for algorithm training, Teledyne FLIR (2018)”) to form a dataset. At the same time, using some data enhancement strategies including rotation, scaling, cropping and contrast adjustment, the original dataset is uniformly cropped to 256×256 image size, generating 130268 image patches. In this way, we can randomly select 70% of the image patches for training and the remaining 30% for testing.

[0133] We use PyTorch 1.10.0 framework to implement the proposed KEDM model and conduct related experiments on two Nvidia GeForce3090 GPUs, an Intel Core i9-10850K CPU and 64GB RAM memory. For the basic diffusion model architecture, the pre-trained VAE encoder encodes both GT images and non-uniformly degraded images into latent space vectors of 64×64 resolution, and sets self-attention layers at the spatial resolutions of 8, 16 and 32 in Denoising-UNet. Consistent with the common training settings of the diffusion model, we also set 1000 forward diffusion time steps for training and 200 backward diffusion time steps for inference, and use the Adam optimizer for optimization. In this process, the batch size, learning rate and iteration number are set to 48, 1e-5 and 1000000 respectively.

[0134] 2. Comparison with other typical methods

[0135] To verify the performance of the proposed algorithm, we compare the designed KEDM with several state-of-the-art infrared image stripe removal methods (respectively from "Y.Cao,MYYang,and C.-L.Tisse,“Effective strip noise removal forlow-textured infrared images based on 1-d guided filtering,”IEEETrans.Circuits Syst.Video Technol.,vol.26,no.12,pp.2176–2188,2015.”, "L.Liu,L.Xu,and H.Fang,“Simultaneous intensity bias estimation and stripe noiseremoval in infrared images using the global and local sparsity constraints,”IEEETrans.Geosci.Remote Sens.,vol.58,no.3,pp.1777–1789,Mar.2020.”, "Z.He,Y.Cao,Y.Dong,J.Yang,Y.Cao,and CLTisse,“Single-image-based nonuniformitycorrection of uncooled long-wave infrared detectors: a deep-learning approach," Appl Opt, vol.57, no.18, pp.D155-D164, Jun 20, 2018.", "Y.Chang, L.Yan, L.Liu, H.Fang, and S.Zhong, "Infrared Aerothermal Nonuniform Correction via DeepMultiscale Residual Network," IEEE Geoscience and Remote Sensing Letters, vol.16, no.7, pp.1120-1124, 2019.", "J.Song,J.-H.Jeong,D.-S.Park,H.-H.Kim,D.-C.Seo,and JCYe, “Unsupervised Denoising for Satellite Imagery Using WaveletDirectional CycleGAN,” IEEE Transactions on Geoscience and Remote Sensing, vol.59, no.8, pp.6823-6839, 2021.”) We used the default parameters of each method to ensure a fair comparison. In terms of quantitative metrics, in order to be consistent with related work, we use peak signal-to-noise ratio (PSNR) and structural similarity index measurement (SSIM) to evaluate the performance of the model. The higher the PSNR and SSIM scores, the better the de-striping effect of the infrared image.

[0136] Table 1 below lists the quantitative comparison results of various methods. During the test, we also used a cubic hybrid degradation model (from "C. Wang, M. Xu, Y. Jiang, G. Zhang, H. Cui, L. Li, and D. Li, "Translution-SNet: A Semisupervised Hyperspectral Image Stripe Noise Removal Based on Transformer and CNN," IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1-14, 2022." and "S. Yang, H. Qin, S. Yuan, X. Yan, and H Japa Rahmani, "DestripeCycleGAN: Stripe Simulation CycleGAN for Unsupervised Infrared Image Destriping," 2024.") to simulate Gaussian noise and uniform stripe noise in infrared images respectively. For Gaussian noise, its standard deviation σ is set to 0.05, and for uniform noise, its mean μ is set to 0.1. The best results in Table 1 are shown in bold font. We can clearly see that in most cases, the KEDM stripe removal results proposed in the present invention are better than those of other algorithms.

[0137] Table 1 PSNR / SSIM values ​​of different methods on three test sets under two stripe noises

[0138]

[0139] It can be seen from Table 1 that the PSNR and SSIM indicators of the KEDM proposed in the present invention are improved. Compared with GF, GLSC, DLS, TSWEU and WCGAN, the average PSNR indicator of KEDM increased by 12.9% (from 35.52 to 40.10), 10.7% (from 36.21 to 40.10), 13.1% (from 35.46 to 40.10), 6.6% (from 37.60 to 40.10) and 2.1% (from 39.27 to 40.10), respectively. For the average SSIM index, KEDM increased by 3.5% (from 0.9558 to 0.9892), 1.9% (from 0.9711 to 0.9892), 2.8% (from 0.9621 to 0.9892), 1.1% (from 0.9787 to 0.9892) and 0.6% (from 0.9837 to 0.9892) respectively. Although the increase of KEDM in SSIM index is limited, its average result has reached about 0.99, and its performance on the three data sets is the best. This also proves that the method proposed in this paper does have a better de-streaking effect on different infrared images.

[0140] Figure 5 Several representative images from CVC, INFRARED and FLIR are shown, and their stripe removal effects under different algorithms. In sub-images (c)-(g), we can observe that some algorithms, such as GF, DLS and TSWEU, have more or less residual stripe noise in the areas corresponding to the green circles, which limits the signal-to-noise ratio of the image. For the results of GLSC and WCGAN, although their residual noise is relatively small, it can be seen from the positions of the green arrows in sub-images (d) and (g) that their stripe removal results will make the local area of ​​the image blurred, and some detailed textures in the scene will be lost due to the excessive smoothing of the algorithm, and even artifacts will appear. In contrast, our KEDM method is the only method that can effectively remove stripe noise while maintaining detailed edges. As shown in sub-image (h), its processing result almost eliminates all stripe noise, and the scene details in the image are basically consistent with the GT image, such as the railings on the street and the Z-shaped texture on the window frame, which are well preserved, thus proving the superior performance of our method.

[0141] 3. Ablation test

[0142] In order to further prove the effectiveness of each module of the KEDM algorithm proposed in the present invention, we take the CVC dataset as an example and conduct an ablation test with uniformly distributed stripe noise of μ=0.1. The corresponding results are shown in Table 2 below. Compared with the original diffusion model (DM), we mainly introduced LDM and DWCM modules in the network structure. Therefore, we first use the same initialization parameters to train the original diffusion model, and do not use the DWCM module. The results are shown in the first row of Table 2. It can be seen that although the original diffusion model has relatively good de-striation indicators in the spatial domain, it is directly diffused on the infrared image of the original size, and the amount of calculation is very large, so its reasoning speed is also the slowest; LDM is used to replace the original DM. When the DWCM module is not used, its various indicators are shown in the second row. The corresponding PSNR and SSIM indicators are similar to those of DM, but the reasoning time is significantly reduced to about 1 / 30 of the DM model. This shows that using LDM instead of DM can indeed greatly improve while ensuring the de-striation performance. Model calculation speed; when we introduce the DWCM module into DM, as shown in the third row, the corresponding PSNR and SSIM indicators are further improved, which shows that the introduction of the DWCM module as a priori constraint can indeed better guide the directional diffusion direction of the model and obtain better de-striking effect; finally, when we combine the LDM with the DWCM module, that is, use the entire KEDM model, the corresponding results are shown in the fourth row, and it can be seen that its PSNR and SSIM indicators both reach the optimal results, and the time consumed is only 1.02s, which is very close to the fastest 0.97s, which also shows that the method of the present invention can achieve a good balance between de-striking effect and computational efficiency.

[0143] Table 1. Comparison of ablation experiment results under different settings (bold indicates the best result)

[0144]

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit the technical solution. Those skilled in the art should understand that those modifications or equivalent substitutions of the technical solution of the present invention that do not depart from the purpose and scope of the technical solution should be included in the scope of the claims of the present invention.

Claims

1. A method for removing stripes from infrared images based on knowledge embedding diffusion model, characterized in that: include: S1: Acquire the infrared image with stripe noise to be processed; S2: Input the infrared image with stripe noise to be processed into the trained stripe removal model, and output the corresponding stripe-free infrared image; The stripe removal model is built based on the potential diffusion model, and the training steps are as follows: S201: Acquire an infrared image y with stripe noise and a corresponding infrared clear image x as training samples; S202: Conditionally encode the infrared image y with stripe noise to obtain a conditional feature map c; encode the infrared clear image x to obtain an initial feature vector z0 of the latent space; S203: Continuously add standard Gaussian noise ε to the initial feature vector z0 of the latent space at T discrete time nodes t ~N(0,I), and obtain the potential feature vector z at time t=T T ; N(·) represents Gaussian distribution, I represents the identity matrix; S204: Using T trained denoising networks combined with conditional feature graphs c, the potential feature vector z starting at time t = T is sequentially denoised. t Perform reverse diffusion denoising until time t = 0 to obtain the potential feature estimate S205: Estimation of latent features by decoder Restore and get the stripe-free infrared image S206: Standard Gaussian noise ε added at each time node t And the predicted noise ε output at the corresponding time of the denoising network θ (z t ,t,c) difference calculation noise loss function; z t represents the intermediate latent feature vector at time t; S207: Based on infrared clear image x and de-streak infrared image The direction prior of the stripe noise is introduced to calculate the prior loss function; S208: Calculate the total loss function through the noise loss function and the prior loss function and reversely optimize the parameters of the denoising network; S209: repeating steps S201 to S208 to iteratively train the denoising network until convergence or reaching a preset number of iterations; After training is complete, the processing steps of the streak removal model are as follows: S211: Encode the infrared image with stripe noise to be processed to obtain the potential feature vector of each time node; S212: Conditionally encode the infrared image with stripe noise to be processed to obtain a conditional feature map; S213: by combining the trained T denoising networks with the conditional feature graph, the potential feature vectors starting from time t=T are subjected to reverse diffusion denoising until time t=0 to obtain a potential feature estimate; S214: restoring the potential feature estimation through the decoder to obtain a stripe-free infrared image; S3: Outputting the outputted de-striation infrared image as the stripe removal result of the infrared image with stripe noise to be processed.

2. The infrared image stripe removal method based on knowledge embedding diffusion model as claimed in claim 1, characterized in that: In step S202 and step S211, encoding is performed using a VAE variational autoencoder.

3. The infrared image stripe removal method based on knowledge embedding diffusion model as claimed in claim 1, characterized in that: In step S202 and step S212, the infrared image with striped noise is conditionally encoded by a conditional encoder.

4. The infrared image stripe removal method based on knowledge embedding diffusion model as claimed in claim 1, characterized in that: In step S203, T discrete time nodes are sampled, and standard Gaussian noise ε is randomly generated at each time node. t ; Starting from time t = 0, the standard Gaussian noise ε corresponding to each moment is t Add to the latent space vector z0 layer by layer until t = T, generating T intermediate latent feature vectors z1 to z T .

5. The infrared image stripe removal method based on knowledge embedding diffusion model as claimed in claim 1, characterized in that: In step S204, T denoising networks corresponding to T time nodes are set; at time t = T, the first denoising network converts the intermediate potential feature vector z T and conditional feature map c as input for reverse diffusion denoising, and output the corresponding potential feature estimate The subsequent denoising network takes the potential feature estimate and conditional feature map c output by the previous denoising network as input for reverse diffusion denoising until time t = 0, and finally obtains the potential feature estimate 6. The infrared image stripe removal method based on knowledge embedding diffusion model as claimed in claim 1, characterized in that: In step S205 and step S214, restoration is performed through the VAE decoder.

7. The infrared image stripe removal method based on knowledge embedding diffusion model as claimed in claim 1, characterized in that: In step S206, the noise loss function is expressed as: L noise =||ε t -ε θ (z t ,t,c)|| 2 ; Where: L noise represents noise loss; ε t represents the added Gaussian noise; ε θ (z t ,t,c) represents the predicted noise output of the denoising network, z t represents the intermediate potential feature vector at time t, and c represents the conditional feature map.

8. The infrared image stripe removal method based on knowledge embedding diffusion model as claimed in claim 1, characterized in that: In step S207, the prior loss function is calculated by the following steps: S2071: Directed wavelet convolution module is used to process the infrared clear image x and the de-striated infrared image Perform wavelet decomposition at multiple scales to obtain multi-scale wavelet decomposition results in different directions; S2072: Extract infrared clear image x and de-streak infrared image The vertical component x of the wavelet at each scale HL and S2073: Through the wavelet vertical component x HL and Calculate the prior loss function; The formula is: Where: L prior is the prior loss.

9. The infrared image stripe removal method based on knowledge embedding diffusion model as claimed in claim 8, characterized in that: In S2071, the directional wavelet convolution module includes four convolution filters, which are defined as follows: Where: f LL is a low-pass filter corresponding to the low-frequency information of the input image; f LH 、f HL and f HH There are three high-pass filters, corresponding to the high-frequency information in the horizontal, vertical and diagonal directions of the input image; For a given infrared image I, the multi-scale wavelet decomposition results in different directions are obtained by using convolution filters, and the expression is: Where: Conv(·) represents the convolution operation, corresponding to 4 discrete wavelet convolutions, which is used to further decompose the low-frequency component of the current (i-1) scale into 4 components at the i scale; They are the output results of the infrared image at the i-th scale in the wavelet low frequency, horizontal, vertical and diagonal directions respectively.

10. The infrared image stripe removal method based on knowledge embedding diffusion model as claimed in claim 1, characterized in that: In step S208, the calculation formula of the total loss function is as follows: THE total =L noise +λL prior ; Where: L total represents the total loss function; λ is an adjustable hyperparameter; L prior is the prior loss; L noise Indicates noise loss.

Images