An infrared image super-resolution reconstruction method based on frequency division guided iteration diffusion

Abstract

The application discloses an infrared image super-resolution reconstruction method based on frequency division guided iterative diffusion, and belongs to the technical field of infrared super-resolution reconstruction, and comprises the following steps: constructing a network model, an up-sampling layer, a Gaussian filter, a high-frequency guiding information encoder, a conditional mapping layer and a pre-trained stable diffusion model; preparing a data set, pre-processing the data set and dividing the data set into three parts of a training set, a verification set and a test set; training the network by using the prepared infrared image data set, and following a frequency division guided iterative training strategy in the whole training process until a preset threshold is reached; fine-tuning the network by using the prepared verification set to obtain a final training model; and solidifying the obtained final model parameters and saving the network. The application adopts high-low frequency separation processing, gradually improves the details and definition of images by using the frequency division guided iterative training strategy, so that the super-resolution reconstruction of real-world low-quality infrared images is realized.

Description

Technical Field

[0001] This invention relates to the field of infrared image super-resolution reconstruction technology, specifically to an infrared image super-resolution reconstruction method based on frequency division-guided iterative diffusion. Background Technology

[0002] Infrared images are widely used in thermal imaging, video surveillance, medical diagnosis, and remote sensing. However, due to limitations in optical components and detector size, infrared images often suffer from blurring and low resolution, far inferior to visible light images. To improve image quality, super-resolution techniques reconstruct high-resolution images to restore details, demonstrating potential in military, security, and surveillance fields. However, infrared images are often accompanied by low contrast, blurring, and noise, limiting practical applications. Most existing infrared super-resolution techniques are designed based on fixed degradation assumptions, leading to decreased reconstruction results when actual degradation does not match, and also exhibiting insufficient accuracy and poor adaptability. Therefore, we propose an infrared image super-resolution method based on frequency division-guided iterative diffusion to address these issues.

[0003] The Chinese patent application, authorized under publication number CN117391938B, entitled "An Infrared Image Reconstruction Method, System, Device, and Terminal," first simulates the low-resolution state of real images using various degradation strategies. It then constructs a super-resolution network comprising a convolutional layer for primary feature extraction and three branches for deep feature extraction. The network is pre-trained using a visible light dataset and optimized using an infrared dataset. Finally, it obtains a high-resolution image through sub-pixel convolution and triple interpolation upsampling. However, this method does not clearly distinguish between low-frequency and high-frequency information during feature extraction and recovery, resulting in insufficient recovery of high-frequency details. Furthermore, this method relies on feature transfer from the visible light dataset, failing to fully represent the characteristics of infrared images. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes an infrared image super-resolution reconstruction method based on frequency division-guided iterative diffusion, which solves the problems of insufficient high-frequency detail restoration and poor low-frequency information enhancement in existing infrared image super-resolution reconstruction methods.

[0005] To achieve the above objectives, the present invention specifically adopts the following technical solution:

[0006] An infrared image super-resolution reconstruction method based on frequency division-guided iterative diffusion includes the following steps.

[0007] Step 1, Construct the network model: The entire super-resolution network consists of an upsampling layer, a Gaussian filter, a high-frequency guided information encoder, a conditional mapping layer, and a pre-trained stable diffusion model.

[0008] Step 2, Prepare the dataset: This invention selects the LLVIP infrared dataset and performs selection and preprocessing on the dataset.

[0009] Step 3: Select the loss function and evaluation index of this super-resolution method: Construct the overall loss function of the super-resolution network to minimize the difference between the generated super-resolution image and the real high-resolution image, and select the evaluation index of this super-resolution method to evaluate the performance of the overall network model.

[0010] Step 4, train the network model: input the training set into the network model for training. The entire training process follows the frequency-guided iterative training strategy. At the same time, to ensure the effect of the pre-trained stable diffusion model, fix the weights of the model and optimize only the parameters of the high-frequency guided information encoder and the conditional mapping layer until the optimal training weights are obtained.

[0011] Step 5, Save the model: After training is complete, solidify the network parameters, store the parameters, weights, code, etc. in the memory, and determine the final infrared image super-resolution reconstruction model.

[0012] Furthermore, the high-frequency information guiding encoder in step 1 is used to encode the input high-frequency information into guiding information for guiding the pre-trained stable diffusion model. The encoder is divided into two branches to extract and enhance the high-frequency information in the spatial and frequency domains respectively, and then the outputs of the two are concatenated to obtain the four-level guiding information.

[0013] Furthermore, the conditional mapping layer in step 1 takes the high-frequency guidance information encoder as input to obtain the potential representation of the guidance information to adapt to the potential diffusion of the stable diffusion model, and transmits the guidance information to the decoder of its U-net network without affecting the weight parameters of the pre-trained stable diffusion model to guide the generation of its reverse process.

[0014] Furthermore, the pre-trained stable diffusion model in step 1 is version V1.5, which has rich prior knowledge and can help the network generate higher quality super-resolution images.

[0015] Furthermore, the overall loss in step 3 consists of pixel loss, perceptual loss, KL divergence loss, control loss, and high-frequency loss, and the evaluation metrics include peak signal-to-noise ratio and structural similarity.

[0016] Furthermore, the frequency-guided iterative training strategy in step 4 decomposes the input infrared image into high-frequency and low-frequency information. The high-frequency information, including details and texture, serves as input to the high-frequency guided information encoder, while the low-frequency information, containing the overall structure of the image, serves as input to the pre-trained stable diffusion model. The high-frequency information is encoded into conditional information to guide the generation of the pre-trained stable diffusion model. During the reverse engineering of the pre-trained stable diffusion model, the low-frequency information at each time step is combined with the high-frequency conditional information to generate a result. The result generated at the nth time step is extracted and used as input to the (n+1)th training round, fed back to the entire network for iteration. This process is repeated to gradually improve the image's details and overall clarity, ultimately generating a high-quality super-resolution image.

[0017] Compared with existing technologies, this invention provides an infrared image super-resolution reconstruction method based on frequency division-guided iterative diffusion, which has the following advantages:

[0018] This invention provides a frequency-division guided iterative training strategy. This strategy decomposes the input image into high-frequency and low-frequency information, which are used as inputs to the high-frequency guided information encoder and the pre-trained stable diffusion model, respectively. It also improves the reverse process of the pre-trained stable diffusion model to achieve an iterative feedback mechanism and gradually optimizes the details and sharpness of the image, ultimately obtaining a high-quality super-resolution image.

[0019] This invention provides a high-frequency perception three-dimensional attention module, which obtains high-frequency components in the input features by introducing a wavelet extraction layer, and adaptively adjusts the high-frequency components through a three-dimensional attention map obtained by the attention mechanism, helping the network to better focus on high-frequency information.

[0020] This invention provides a conditionally guided diffusion network model, which uses conditional information generated by a high-frequency guided information encoder to constrain a pre-trained stable diffusion model, thus solving the problem of inconsistent generation results of the pre-trained stable diffusion model and achieving more realistic super-resolution image reconstruction. Attached Figure Description

[0021] Figure 1 A flowchart of an infrared image super-resolution reconstruction method based on frequency division-guided iterative diffusion provided by the present invention;

[0022] Figure 2 A schematic diagram of a high-frequency conditionally guided diffusion network model structure provided by the present invention;

[0023] Figure 3 A schematic diagram of the high-frequency guidance information encoder structure provided by the present invention;

[0024] Figure 4 This is a schematic diagram of the frequency domain extraction module provided by the present invention.

[0025] Figure 5 This is a schematic diagram of the spatial domain extraction module provided by the present invention.

[0026] Figure 6 A schematic diagram of the extended fusion module structure provided by the present invention.

[0027] Figure 7 A schematic diagram of the high-frequency sensing three-dimensional attention module structure provided by the present invention.

[0028] Figure 8 This is a schematic diagram of the conditional mapping layer structure provided by the present invention.

[0029] Figure 9 This provides a comparative diagram of evaluation indicators between the method of this invention and existing technologies. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] Example 1

[0032] like Figure 1 The flowchart shown in this embodiment of the invention provides a method for super-resolution reconstruction of infrared images based on frequency division-guided iterative diffusion, which specifically includes the following steps:

[0033] Step 1, Construct the network model: such as Figure 2The diagram shows the structure of a high-frequency conditional guided diffusion network model, which includes: an upsampling layer, a Gaussian filter, a high-frequency guided information encoder, a conditional mapping layer, and a pre-trained stable diffusion model. The high-frequency guided information encoder extracts and enhances features from the input high-frequency image in both the frequency and spatial domains. The frequency domain feature extraction branch consists of four frequency domain extraction blocks with local residual connections; the spatial domain feature extraction branch also consists of four spatial domain extraction blocks with local residual connections. Each frequency and spatial extraction block corresponds one-to-one and is divided into four levels. The features processed at each level are concatenated to obtain the four levels of guided information. Finally, a 1×1 convolutional layer is used to process the last stage of concatenated features to obtain the processed high-frequency feature map. The conditional mapping layer consists of four 1×1 convolutional layers, a VAE encoder, and four zero convolutional layers. The 1×1 convolutional layers are used to adjust the number of channels for each stage of conditional information, while the VAE encoder is used to map the conditional information to the latent space to adapt to the latent diffusion process of the stable diffusion model. Subsequently, the conditional information is mapped to the U-net decoder of the stable diffusion model through the zero convolutional layer to guide the generation of high-quality images. The pre-trained stable diffusion model is version V1.5, which utilizes its rich prior knowledge to achieve the generation of high-quality super-resolution images.

[0034] Step 2, Prepare the dataset: This method uses the LLVIP dataset and selects and preprocesses the dataset;

[0035] Step 3: Select the loss function and evaluation metrics for this super-resolution method: Based on the network model in Step 1, select a loss function to minimize the difference between the super-resolution image reconstructed by the network and the real high-resolution image, and select an evaluation metric to evaluate the network performance; This invention uses pixel loss, perceptual loss, KL divergence loss, control loss, and high-frequency loss; pixel loss is used to ensure that the super-resolution image is as close as possible to the real high-resolution image at the pixel level; perceptual loss is used to improve the visual quality of the super-resolution image, making the super-resolution image more perceptually similar to the real image; KL divergence loss is used to normalize the latent distribution of the VAE encoder, making the latent spatial structure more reasonable; control loss is used to ensure the conditional generation of the pre-trained stable diffusion model; high-frequency loss is used to guide and strengthen the high-frequency guided encoder, making it closer to the real image in texture and contour. During training, peak signal-to-noise ratio and structural similarity are used as evaluation metrics;

[0036] Step 4, Training the Model: Input the training set from the dataset constructed in Step 2 into the network model to train the network model. The entire model training process follows a frequency-guided iterative training strategy, such as... Figure 2The diagram illustrates the training process of a frequency-guided iterative training strategy. This strategy decomposes the input infrared image into high-frequency and low-frequency information. The high-frequency information includes details and textures, while the low-frequency information contains the overall structure of the image. The high-frequency information is encoded into conditional information to guide the generation of a pre-trained stable diffusion model. During the reverse process of the stable diffusion model, the low-frequency features at each time step are combined with the high-frequency conditional information to generate the result. The result generated at the nth time step is extracted and used as the input for the (n+1)th training round, feeding back into the entire network for iteration until a training threshold is reached. This gradually improves the image's details and overall clarity, ultimately generating a high-quality super-resolution image. To ensure the effectiveness of the stable diffusion model, its weight parameters remain fixed; only the parameters of the high-frequency guided information encoder and the conditional mapping layer are optimized until the optimal training weights are obtained.

[0037] Step 5: Save the model. After training is complete, solidify the network parameters, store the parameters, weights, code, etc. in the memory, and determine the final infrared image super-resolution reconstruction model.

[0038] Example 2

[0039] like Figure 1 The flowchart shown is a method for super-resolution reconstruction of infrared images based on frequency division-guided iterative diffusion infrared images, according to an embodiment of the present invention. The method specifically includes the following steps:

[0040] Step 1, Construct the network model: such as Figure 2 The high-frequency conditionally guided diffusion network model constructed in this invention is shown, specifically including: an upsampling layer, a Gaussian filter, a high-frequency guided information encoder, a conditional mapping layer, and a pre-trained stable diffusion model;

[0041] The structure of the high-frequency guidance information encoder is as follows: Figure 3 As shown, the high-frequency guided information encoder utilizes both spatial and frequency domains to perform multi-level feature extraction and enhancement of the input high-frequency features, ensuring that the generated conditional information retains rich high-frequency details. In the encoder, each of the spatial and frequency domain feature extraction branches consists of four feature extraction blocks connected by local residuals. Each frequency domain extraction block corresponds one-to-one with a spatial domain extraction block; therefore, the entire encoder can be divided into four levels. At each level, spatial and frequency domain features are concatenated to generate fused fourth-level conditional information to guide the stable diffusion model in generating high-quality images. Finally, the fourth-level fused features are processed by a 1×1 convolutional layer, outputting the enhanced high-frequency feature map.

[0042] Frequency domain extraction block structure as follows Figure 4As shown, firstly, the high-frequency features of the input are subjected to inverse Fourier transform to generate a spectrogram. Then, a high-low frequency mask is used to separate the high-frequency and low-frequency information in the spectrogram. Next, high- and low-frequency information is extracted through 1×1 convolution and an L-shaped activation function. For the spectrogram S(f) after the low-frequency mask, x ,f y The high-frequency information is enhanced by using an adaptive moving weighted average algorithm.

[0043] The spectrogram is divided into N×M blocks, each denoted as B. i,j (f x ,f y ), where i and j represent the block and column indexes, respectively, and each block needs to be assigned an enhancement factor A. i,j (f x ,f y To reasonably constrain the intensity of high-frequency information enhancement, the enhancement factor is set based on the relationship between each block and the center frequency (f). cx ,f cy The distance of ) and the enhancement coefficient A i,j (f x ,f y ) is represented as:

[0044]

[0045] Where α represents the control reinforcement strength coefficient, d i.j (f x ,f y ) represents the frequency component within the block (f x ,f y ) to center frequency (f cx ,f cy The normalized distance of σ i,j (f x ,f y The local variance within the block is used to adaptively control the enhancement coefficient.

[0046] After calculating the adaptive enhancement coefficient A for each block i,j (f x ,f y After that, the enhancement results of each block are weighted to ensure a smooth transition between each block, and finally the enhanced high-frequency spectral components are obtained. (f x ,f y Its form of expression is:

[0047]

[0048] Where k i,j (f x ,f y) represents the weighting coefficient between each block, S i,j (f x ,f y ) indicates the original spectrogram in block B i,j (f x ,f y The high-frequency spectral components within ).

[0049] The enhanced high-frequency component is multiplied by the original high-frequency spectrum component to obtain the enhanced high-frequency information. This enhanced high-frequency information is then added to the low-frequency information extracted by 1×1 convolution and L-type activation function after high-frequency masking to generate the enhanced spectrum. Finally, the spectrum is transformed from the frequency domain back to the spatial domain by inverse Fourier transform to obtain the output of the frequency domain extraction block.

[0050] The structure of the spatial domain extraction block is as follows Figure 5 As shown, the high-frequency features of the input After two downsampling operations, the results were obtained. and Feature extraction is performed on high-frequency features in three different dimensions. Each feature extraction branch includes a multi-scale expansion fusion block, a wavelet three-dimensional intention module, and a 1×1 convolutional layer.

[0051] The structure of the multi-scale extended fusion block is as follows: Figure 6 As shown, it consists of three dilated convolutional layers and a concatenation operation. The three dilated convolutional layers are composed of dilated convolutions with a kernel size of 3×3 and dilation rates of 2, 3 and 4 respectively, and L-shaped activation functions to achieve feature extraction at different scales. Finally, the features extracted at the three different scales are concatenated as the output of the dilated fusion block.

[0052] The structure of the high-frequency perception 3D attention module is as follows: Figure 7 The structure shown includes spatial attention, channel attention, and wavelet extraction layers, with the input feature map... These three parts are input in parallel. Spatial attention: Global max pooling and global average pooling are performed on the input feature map to generate a spatial attention map. Channel attention: Perform global max pooling and global average pooling on the input feature map along its channel dimensions, and feed the results into a fully connected layer to generate a channel-averaged feature map. and channel maximum feature map Wavelet Extraction Layer: The input feature map is decomposed into low-frequency and high-frequency components using discrete wavelet transform. A first convolutional layer (3×3 kernel with R-type activation function) integrates the high-frequency components to obtain a high-frequency feature map. A second convolutional layer (3×3 kernel with R-type activation function) then transforms the high-frequency feature map into a high-frequency spatial feature map. The high-frequency feature map is converted into a high-frequency channel feature map using an average pooling layer. The high-frequency spatial feature map and the spatial attention map are concatenated by channels, and then processed through a fourth convolutional layer (with a 3×3 kernel and an R-type activation function) to generate a high-frequency spatial attention map. Pixel-level addition is performed between the high-frequency channel feature map and the average and maximum features of the channel attention to generate the high-frequency channel attention map. By using a broadcast mechanism, the high-frequency spatial attention map is multiplied with the high-frequency channel attention map to generate a three-dimensional attention map. The 3D attention weight map is obtained using the Sigmoid activation function, and then multiplied with the input features through a residual connection to obtain the output of the high-frequency perceptual 3D attention module.

[0053] The outputs of the high-frequency perception 3D attention modules in all three branches are processed by a 1×1 convolutional layer to obtain three features, namely... and right Use deconvolution operations and combine them with features. Features are obtained by performing pixel-level addition operations. Features Use deconvolution operations and combine them with features. Pixel-level addition is used to obtain features Finally, the features and Perform pixel-level addition to obtain the output of the spatial domain extraction block.

[0054] The structure of the condition mapping layer is as follows Figure 8 As shown, the model includes four 1×1 convolutional layers (1×1 kernel size, with R-type activation function), a VAE encoder, and four 0 convolutional layers (1×1 kernel size, weights initialized to 0). The four 1×1 convolutional layers receive four levels of guidance information generated by the high-frequency guidance information encoder. After adjusting the number of channels, this information is used as input to the VAE encoder to generate a latent representation of the four levels of guidance information. Each latent representation is then used as input to the four 0 convolutional layers, injecting conditional information while maintaining no impact on the parameters of the pre-trained stable diffusion model. This allows the conditional information to act on the four upsampling layers of the model encoder.

[0055] This invention uses a pre-trained stable diffusion model of version V1.5, which has a wealth of prior knowledge due to its large-scale data training, to help the network generate clearer and more realistic high-resolution images in infrared image super-resolution reconstruction tasks.

[0056] To ensure the robustness of the network, introduce more nonlinear factors, and retain more image information, this invention employs four activation functions: R-type activation function, L-type activation function, and S-type activation function, defined as follows:

[0057]

[0058] Step 2, Prepare the dataset: The LLVIP dataset contains 17,088 infrared images covering 10 scene categories. We selected 4,690 infrared images with a size of 1080 pixels × 720 pixels and performed image augmentation to expand the dataset to 14,570 infrared images with a size of 256 × 256 pixels. Using a 6:2:2 ratio, the dataset was divided into a training set, a validation set, and a test set. Adaptive histogram equalization was then performed on the training set to improve the image quality.

[0059] Step 3: Select the loss function and evaluation metric for this super-resolution method: Based on the network model in Step 1, select the loss function and evaluation metric. The high-frequency guided information encoder uses high-frequency loss, the pre-trained stable diffusion model uses control loss, the VAE encoder uses KL divergence loss, and the entire super-resolution network model uses... Pixel loss and perceptual loss;

[0060] High-frequency guidance information encoders use high-frequency loss To minimize the generated high-frequency feature I H Compared to real high-resolution images I HR The difference between the high-frequency information extracted by the Laplace operator Δ is specifically expressed as follows:

[0061]

[0062] Pre-trained stable diffusion models use control loss This allows for the generation of high-quality images under the guidance information c at each time step t, specifically expressed as:

[0063]

[0064] Where ε represents the noise added to the input low-frequency information during the forward process at each time step, ε θ (x t ,t,c) represents the noise predicted by the model under the control of the guiding information at each time step.

[0065] The VAE encoder uses KL divergence loss. The divergence loss is used to measure the difference between the latent distribution q(z|x) and the prior distribution p(z). The latent distribution q(z|x) is the distribution of the latent variable z given the guiding information x, while the prior distribution p(z) is a hypothetical distribution used to define the overall structure of the latent space. The mathematical expression is:

[0066]

[0067] Pixel loss is used to measure the generated super-resolution I SR Compared to real high-resolution images I HR The difference between pixels is expressed mathematically as follows:

[0068]

[0069] Perceptual loss is achieved through a pre-trained convolutional neural network. The generated super-resolution image I SR Compared to real high-resolution images I HR High-level feature extraction is performed, and the generated image I is evaluated by difference. SR With target image I HR The similarity between the images makes the network-generated images more consistent with human perception. The specific mathematical expression is:

[0070]

[0071] in Indicates pre-trained network The i-th feature layer in the model, where N represents the number of feature layers selected.

[0072] This invention combines the above-mentioned losses, and the overall loss is as follows:

[0073]

[0074] Where λ1, λ2, λ3, λ4 and λ5 represent the weights of different losses in the overall loss. Optimizing the overall loss helps the network improve reconstruction accuracy and generate more realistic super-resolution images.

[0075] Peak signal-to-noise ratio (RNSR) and similarity contrast ratio (SSIM) are selected as the evaluation metrics of this invention, and are defined as follows:

[0076]

[0077] Wherein, RNSR represents peak signal-to-noise ratio, MAX Iσ represents the maximum possible value of an image pixel, MSE represents the mean squared error used to measure the difference between the original image and the compressed or reconstructed image. x σ represents the contrast of image x. y C2 represents the contrast of image y, and C2 is a constant to prevent the denominator from being 0.

[0078] Step 4, Training the network model: The entire network model training process follows the frequency division-guided iterative training strategy proposed in this invention, such as... Figure 2 The diagram illustrates the training process of the frequency division-guided iterative training strategy. First, the input infrared image is processed through a Gaussian filter to extract its low-frequency features, which are then subtracted from the original image to obtain the high-frequency features. This process can be represented as:

[0079]

[0080] I H =II L Where I represents the input infrared image, I H Represents the high-frequency features of an image, including details and texture, I L Represents the low-frequency features of an image, including the overall structure of the image. This indicates a Gaussian filter operation. For the filter parameters, the standard deviation is set to 2, and the kernel size is 11×11. This operation smooths the low-frequency structure while ensuring good separation of high-frequency details.

[0081] The extracted high-frequency and low-frequency features are used as inputs to the high-frequency guidance information encoding and the stable diffusion model, respectively. The high-frequency guidance information encoder extracts and enhances key features in the high-frequency information, encoding them as conditional information. This process can be represented as:

[0082] C H =ε(I H ),

[0083] Where C H ε(·) represents the conditional information generated by the high-frequency guided information encoder through the extracted high-frequency features, and represents the high-frequency guided information encoding operation.

[0084] Low-frequency features are used as input to a pre-trained stable diffusion model to generate images through a back-diffusion process. At each time step t, the input low-frequency features are extracted by the encoder in the U-net network of the stable diffusion model. During the decoding phase, these features, guided by conditional information, are used by the decoder to reconstruct the generated image at that time step. This process follows a normal distribution and can be represented as:

[0085]

[0086] Among them I S,t+1 This represents the image generated at time step t+1 representing the reverse process of the stable diffusion model. I represents the reverse diffusion process in a steady-state diffusion model. L,t This represents the low-frequency features of the input model at time step t.

[0087] After each time step of the stable diffusion model completes the reverse process, it extracts the result generated at that time and uses it again as the new input to the entire super-resolution network to iterate the training process. Therefore, the high- and low-frequency feature extraction process can be represented as:

[0088]

[0089] in This represents the image generated by the stable diffusion model during the nth iteration of the training process. Indicates network When used as new input, the low-frequency features extracted in the (n+1)th iteration This represents the high-frequency features extracted during the (n+1)th iteration.

[0090] The high-frequency guidance information encoding process during the (n+1)th iteration can be represented as:

[0091]

[0092] in This represents the high-frequency features extracted by the network in the (n+1)th iteration. Conditional information generated by the high-frequency guided encoder.

[0093] The reverse process of the stable diffusion model in the (n+1)th iteration can be represented as:

[0094]

[0095] in This represents the result generated by the reverse process of the stable diffusion model during the (n+1)th iteration.

[0096] In summary, the entire training process of this strategy can be represented as follows:

[0097]

[0098] in This represents the super-resolution image generated by the entire network after N iterations.

[0099] Training is complete when the training loss reaches a set threshold. During training, the Adam optimizer continuously optimizes the network parameters of the high-frequency guided information encoder and conditional mapping layer. The parameters of the pre-trained stable diffusion model are fixed and do not participate in updates. The learning rate is set to 1×10. -4 After training, the model is tested using a test set and evaluated using the evaluation metrics from step 3.

[0100] Step 5, Save the model: After the network model is trained, solidify all trained parameters and determine the final super-resolution network. If performing infrared image super-resolution tasks, the low-resolution infrared image can be directly input into the trained end-to-end network model to obtain the final super-resolution image.

[0101] Among them, the implementation of convolution, deconvolution, activation function, concatenation operation, upsampling and Gaussian filtering are algorithms well known to those skilled in the art, and the specific processes and methods can be found in the relevant textbooks or technical literature.

[0102] This invention designs a frequency-division-guided iterative diffusion infrared image super-resolution method and applies it to the super-resolution task of infrared images. It realizes end-to-end network input and output, and effectively solves the long-standing problem of detail loss in infrared image super-resolution. This makes the infrared super-resolution task more effective. Under the same conditions, the relevant indicators of the super-resolution image obtained by existing methods further verify the feasibility and superiority of this method.

[0103] A comparative diagram of evaluation indicators between existing technologies and the method proposed in this invention is shown below. Figure 9 As shown in the figure, when the reconstruction magnification is 2 and 4, the method proposed in this invention has a higher peak signal-to-noise ratio and structural similarity than existing methods. These indicators demonstrate that the method proposed in this invention performs better than existing methods, and the reconstructed high-resolution images are clearer.

[0104] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for super-resolution reconstruction of infrared images based on frequency division-guided iterative diffusion, characterized in that, The specific steps are as follows: Step 1, Construct the network model: The entire super-resolution network consists of an upsampling layer, a Gaussian filter, a high-frequency guided information encoder, a conditional mapping layer, and a pre-trained stable diffusion model; Step 2, Prepare the dataset: Use the LLVIP dataset and select and preprocess the dataset; Step 3, Select loss function and evaluation metrics: Based on the network model in Step 1, select a loss function to minimize the difference between the super-resolution image reconstructed by the network and the real high-resolution image, and select evaluation metrics to evaluate the network performance; use pixel loss, perceptual loss, KL divergence loss, control loss and high-frequency loss; select peak signal-to-noise ratio and structural similarity as evaluation metrics; Step 4, train the network model: input the training set from the dataset constructed in Step 2 into the network model to train the network model. The entire training process follows the frequency-guided iterative training strategy. At the same time, in order to ensure the effect of the pre-trained stable diffusion model, its weight parameters are kept fixed, and only the parameters of the high-frequency guided information encoder and the conditional mapping layer are optimized until the optimal training weights are obtained. Step 5: Save the model. After training is complete, solidify the network parameters, store the parameters, weights, and code in the memory, and determine the final infrared image super-resolution reconstruction model. The frequency-guided iterative training strategy in step 4 first decomposes the input infrared image into high-frequency information and low-frequency information. The high-frequency information contains details and texture information, while the low-frequency part contains the overall structure of the image. The high-frequency information is used as the input of the high-frequency guided information encoder, and the low-frequency information is used as the input of the pre-trained stable diffusion model. The high-frequency information is encoded into conditional information to guide the generation of the pre-trained stable diffusion model. In the reverse process of the stable diffusion model, the low-frequency features at each time step are combined with the high-frequency conditional information to generate results. The result generated at the nth time step is extracted and used as the input of the (n+1)th round of training and fed back to the entire network for iteration until the training threshold is reached.

2. The infrared image super-resolution reconstruction method based on frequency division-guided iterative diffusion according to claim 1, characterized in that, The high-frequency guidance information encoder in step 1 extracts and enhances the features of the input high-frequency information through two dimensions: frequency domain and spatial domain. The frequency domain feature extraction branch consists of four frequency domain extraction blocks with local residual connections; The spatial feature extraction branch also consists of four spatial extraction blocks with local residual connections; each frequency domain and spatial extraction block corresponds one-to-one and is divided into four levels. The features processed at each level are spliced ​​together to obtain four levels of guiding information.

3. The infrared image super-resolution reconstruction method based on frequency division-guided iterative diffusion according to claim 1, characterized in that, The spatial extraction block in step 1 consists of a downsampling operation, three multi-scale expansion and fusion blocks, three high-frequency perceptual 3D attention modules, three 1×1 convolutional layers, and three deconvolutional layers.

4. The infrared image super-resolution reconstruction method based on frequency division-guided iterative diffusion according to claim 3, characterized in that, The high-frequency sensing 3D attention module includes spatial attention, channel attention, and a wavelet extraction layer; spatial attention generates a spatial attention map through global pooling; channel attention generates a channel feature map through global pooling and a fully connected layer. The wavelet extraction layer uses discrete wavelet transform to decompose the feature map, and the high-frequency components are processed by convolution and average pooling layers to generate high-frequency spatial and high-frequency channel feature maps.

5. The infrared image super-resolution reconstruction method based on frequency division-guided iterative diffusion according to claim 1, characterized in that, High-frequency spatial features are fused with spatial attention to generate a high-frequency spatial attention map, and high-frequency channel features are fused with channel features to generate a high-frequency channel attention map. Finally, a three-dimensional attention map is generated by multiplying the two features through a broadcast mechanism, and the result is output after activation function and residual connection.

Images