Image restoration method for infrared simple lens imaging system with wide signal-to-noise ratio range

Abstract

This invention discloses an image restoration method for a wide signal-to-noise ratio range infrared simple lens imaging system, belonging to the field of infrared image processing. The steps are as follows: S1: Collect the infrared simple lens imaging dataset as the training dataset for the restoration model; S2: Construct a Wavelet-ProxNet model as the restoration model, which adopts an asymmetric encoder-decoder structure, including an encoder and a decoder; S3: Establish a composite loss function as a constraint, and train the restoration model using the training dataset; S4: For the image to be restored, generate the restored image from the trained restoration model. This invention provides a diffractive lens infrared image reconstruction network model and its reconstruction method that combine high reconstruction quality, strong noise generalization, and low computational complexity, promoting the practical application and lightweight development of diffractive lens infrared imaging technology.

Description

Technical Field

[0001] This invention relates to the field of infrared image processing technology, and in particular to an image restoration method for a wide signal-to-noise ratio range infrared simple lens imaging system. Background Technology

[0002] Traditional infrared optical systems correct aberrations through a combination of multiple lenses, resulting in large system size, heavy weight, and high cost, making them unsuitable for lightweight platforms such as drones, microsatellites, and portable devices. In existing simple infrared lens imaging systems based on diffractive lenses, transferring some aberration correction to image post-processing has become an effective method for simplifying the optical system. However, this imaging method requires image post-processing for image restoration. Existing diffractive lens image reconstruction methods mainly fall into three categories: physical-driven methods, data-driven methods, and hybrid-driven methods. Physical-driven methods, such as inverse filtering, Wiener filtering, and Richardson-Lucy iterative deconvolution, are based on the physical model of the imaging system, are sensitive to noise, easily amplify noise during reconstruction, and experience performance degradation under spatially varying and blurred conditions. Data-driven methods can be divided into those based on deep learning models such as convolutional neural networks, Transformers, and Mamba. These methods suffer from insufficient model generalization ability, large model parameter count, and high computational complexity, making them difficult to run in real-time on embedded devices. Hybrid-driven methods, such as PnP and deep unrolling, can combine the generalization of physical models with the noise robustness of deep learning, but they have high computational resource requirements and are difficult to deploy on lightweight platforms. Summary of the Invention

[0003] The purpose of this invention is to provide an image restoration method for a simple infrared lens imaging system with a wide signal-to-noise ratio range, and to provide a diffractive lens infrared image reconstruction network model and reconstruction method that have high reconstruction quality, strong noise generalization and low computational complexity, so as to promote the practical application and lightweight development of diffractive lens infrared imaging technology.

[0004] To achieve the above objectives, the present invention provides an image restoration method for a wide signal-to-noise ratio range infrared simple lens imaging system, comprising the following steps:

[0005] S1: Collect a dataset of infrared simple lens imaging, and use clear and degraded images as training datasets for the restoration model;

[0006] S2: Construct the Wavelet-ProxNet model as the restoration model. The restoration model adopts an asymmetric encoder-decoder structure, including an encoder and a decoder. The encoder includes an encoding module and a strided convolutional downsampling module, and the decoder includes a decoding module and an upsampling module.

[0007] S3: Establish the composite loss function As a constraint, the restoration model is trained using the training dataset;

[0008] S4: For the image to be restored, the restored image is generated by the trained restoration model.

[0009] Preferably, in S1, the process of obtaining the training dataset is as follows:

[0010] We collected high-quality open-source infrared datasets, performed degradation calibration on a simple infrared lens imaging system, convolved the calibrated point spread function (PSF) with the high-quality infrared dataset, added dynamic Gaussian noise to simulate detector noise, and obtained degraded images. We then combined the degraded images and the corresponding clear images to form the training dataset for the restoration model.

[0011] Preferably, in S2, the encoder's encoding module is as follows:

[0012] The encoding module adopts the MetaFormer structure, including a region-biased convolutional attention module (ABCAM) and a wavelet transform-based feedforward module (WaveletMLP). The ABCAM module consists of a 1×1 region-biased convolution, a 3×3 region-biased convolution, a SimpleGate, a Simple Channel Attention (SCA) mechanism, and a 1×1 region-biased convolution connected in sequence. The WaveletMLP module consists of a Discrete Wavelet Transform (DWT) module, a 1×1 convolution, a LeakyReLU module, a 1×1 convolution, an inverse discrete wavelet transform, a 3×3 convolution, a LeakyReLU module, and a 1×1 convolution connected in sequence for the input features. Output after encoding module as follows:

[0013] ;

[0014] Here, WaveletMLP represents a wavelet transform-based feedforward module. It is layer normalization.

[0015] Preferably, in S2, the encoder's strided convolution downsampling module is as follows: using strided convolution with a stride of 2, after the input features are processed by the strided convolution downsampling module, the feature space resolution is reduced to 1 / 2 of the input size, and the number of channels becomes twice that of the input.

[0016] Preferably, in S2, the decoding module in the decoder is as follows:

[0017] The decoding module includes a feature domain gradient descent module (GDM) and a near-end mapping module (PMM); the GDM uses residual blocks to predict the degradation matrix and its transpose. The iterative formula for gradient descent is obtained. as follows:

[0018] ;

[0019] In the above formula, It is the first Level decoder Sub-iteration decoder features, when hour For the first Level output Upsampling, Indicates the step size. This represents the predicted residual block of the degenerate matrix transpose. It is a degenerate matrix prediction residual block. It is the first Level encoder features;

[0020] The Proximal Mapping Module (PMM) uses the NAFBlock module, and its output formula is as follows:

[0021] ;

[0022] In the above formula, CAM represents the convolutional attention module, LayerNorm represents layer normalization, and MLP represents the feedforward module. Indicates intermediate calculation quantities, set This is the upper limit of iterations, when the number of iterations... At that time, the gradient descent iterative formula and the output formula of the proximal mapping module are updated respectively. and .

[0023] Preferably, in S2, the upsampling module in the decoder consists of a subpixel convolution module and Conv2d, used to adjust the feature resolution and channels. After the input features are downsampled by the module, the feature space resolution becomes twice the input size, and the number of channels becomes half of the input.

[0024] Preferably, in S3, the process of establishing the composite loss function is as follows:

[0025] Establish a composite loss function:

[0026] ;

[0027] In the above formula, Represents the main loss function. Represents the denoising loss function. and These are the weighting coefficients;

[0028] Main loss function The loss includes L1 loss, L2 loss, total variational loss, and structural similarity loss, as shown in the following formulas:

[0029] ;

[0030] In the above formula, , , and These are the weight coefficients for the four loss functions. The restored image. For a clear image, Indicates L1 loss, Indicates L2 loss, This represents the total variational loss. Represents structural similarity loss;

[0031] Denoising loss function This includes L1 loss, L2 loss, and total variational loss;

[0032] ;

[0033] In the above formula, , and These are the weight coefficients for the three loss functions. This indicates an 8x reduction in resolution. This represents the low-resolution output of the encoding network. For a clear image, For degenerate convolution kernels, It is a convolution operator.

[0034] Preferably, in S3, the training process is as follows:

[0035] Using the composite loss function As a constraint, the restoration model is trained using degraded images from the training dataset. The encoder extracts features at different resolutions, and these extracted features are fed into a convolutional layer to supervise the denoising capabilities of the encoding module. The multi-level features extracted by the encoder are then further deblurred by the decoder to generate the restored image. During training, the batch size is set to 1, and the Adam optimizer is used to update the parameters. Set it to 0.9. Set it to 0.999. This represents the first-order momentum coefficient. This represents the second-order momentum coefficient. The training process is divided into two stages: the first stage, the learning rate is calculated using cosine annealing from... Upgraded to In the second stage, a cosine annealing strategy is used to reduce the learning rate from... Gradually decrease to .

[0036] Therefore, the image restoration method for a wide signal-to-noise ratio range infrared simple lens imaging system described above has the following advantages:

[0037] 1) This invention performs image denoising in the wavelet domain and depth unfolding deblurring in the feature domain, which can give full play to the advantages of different technologies. By using wavelet domain denoising and combining it with feature domain optimization, it outperforms existing mainstream methods in terms of PSNR, SSIM, MUSIQ and other indicators.

[0038] 2) Strong noise robustness: Feature separation is performed in the wavelet domain to enhance the model's ability to generalize to various noise levels;

[0039] 3) High computational efficiency: The model has only 3.25M parameters and 60.06 GFLOPs of computation, resulting in fast inference speed and suitability for embedded deployment;

[0040] 4) Lightweight structure: By using designs such as region bias convolution and feature domain expansion, the complexity of the model is greatly reduced while ensuring performance.

[0041] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0042] Figure 1 This is a schematic diagram illustrating the principle of the restoration model in the image restoration method of a wide signal-to-noise ratio range infrared simple lens imaging system of the present invention;

[0043] Figure 2 This is a schematic diagram illustrating the principle of standard convolution and region-biased convolution in the image restoration method of a wide signal-to-noise ratio range infrared simple lens imaging system of the present invention.

[0044] Figure 3 This paper compares the restoration performance of the restoration model in the image restoration method of the infrared simple lens imaging system with that of the present invention with that of existing methods on a simulation dataset, with a noise variance of 0.0075.

[0045] Figure 4 This paper compares the restoration performance of the restoration model in the image restoration method of the infrared simple lens imaging system with that of the present invention on a simulation dataset, with a noise variance of 0.030.

[0046] Figure 5 This paper compares the restoration performance of the restoration model in the image restoration method of a wide signal-to-noise ratio infrared simple lens imaging system of the present invention with that of existing methods on real-world datasets. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Specific model specifications need to be selected and determined according to the actual specifications of the device, etc. The specific selection calculation method adopts existing technology in the art, and therefore will not be described in detail.

[0048] Example

[0049] like Figures 1-2 As shown, this invention provides an image restoration method for a wide signal-to-noise ratio range infrared simple lens imaging system, comprising the following steps:

[0050] S1: Collect the dataset of infrared simple lens imaging, and use the clear image and the degraded image as the training dataset of the restoration model; In this embodiment, the working band of infrared simple lens imaging is 3.7μm~4.8μm, the focal length is 54mm, and the detector pixel size is 15μm; Degrade calibration is performed on the infrared simple lens imaging system, and the calibrated point spread function (PSF) is convolved with the high-quality infrared dataset, and dynamic Gaussian noise is added to simulate detector noise to obtain the degraded image. The degraded image and the corresponding clear image are used to form the training dataset of the restoration model;

[0051] S2: Construct the Wavelet-ProxNet model as the restoration model. The restoration model adopts an asymmetric encoder-decoder structure, including an encoder and a decoder. The encoder includes an encoding module and a strided convolutional downsampling module. The encoding module is responsible for feature denoising of the input degraded image in the wavelet domain, and the downsampling module completes feature downsampling. The specific encoding module of the encoder is as follows:

[0052] The encoding module adopts the MetaFormer structure, including a region-biased convolutional attention module (ABCAM) and a wavelet transform-based feedforward module (WaveletMLP). The ABCAM module consists of a 1×1 region-biased convolution, a 3×3 region-biased convolution, a SimpleGate, a Simple Channel Attention (SCA) mechanism, and a 1×1 region-biased convolution connected in sequence. The WaveletMLP module consists of a Discrete Wavelet Transform (DWT) module, a 1×1 convolution, a LeakyReLU module, a 1×1 convolution, an inverse discrete wavelet transform, a 3×3 convolution, a LeakyReLU module, and a 1×1 convolution connected in sequence for the input features. Output after encoding module as follows:

[0053] ;

[0054] Here, WaveletMLP represents a wavelet transform-based feedforward module. It is layer normalization.

[0055] The encoder's strided convolution downsampling module is as follows: using strided convolution with a stride of 2, after the input features are processed by the strided convolution downsampling module, the feature space resolution is reduced to 1 / 2 of the input size, and the number of channels becomes twice that of the input.

[0056] The decoder consists of a decoding module and an upsampling module. The decoding module is responsible for performing gradient descent in the feature domain to improve the sharpness of the denoised low-resolution image, while the upsampling module is used to adjust the feature resolution and channels. The decoding module in the decoder is detailed below:

[0057] The decoding module includes a feature domain gradient descent module (GDM) and a near-end mapping module (PMM); the GDM uses residual blocks to predict the degradation matrix and its transpose. The iterative formula for gradient descent is obtained. as follows:

[0058] ;

[0059] In the above formula, It is the first Level decoder Sub-iteration decoder features, when hour For the first Level output Upsampling, Indicates the step size. This represents the predicted residual block of the degenerate matrix transpose. It is a degenerate matrix prediction residual block. It is the first Level encoder features;

[0060] The Proximal Mapping Module (PMM) uses the NAFBlock module, and its output formula is as follows:

[0061] ;

[0062] In the above formula, CAM represents the convolutional attention module, LayerNorm represents layer normalization, and MLP represents the feedforward module. Indicates intermediate calculation quantities, set This is the upper limit of iterations, when the number of iterations... At that time, the gradient descent iterative formula and the output formula of the proximal mapping module are updated respectively. and .

[0063] The upsampling module in the decoder consists of a subpixel convolution module and Conv2d, which are used to adjust the feature resolution and channels. After the input features are downsampled, the feature space resolution becomes twice the input size, and the number of channels becomes half of the input.

[0064] S3: Establish the composite loss function As a constraint, the restoration model is trained using the training dataset;

[0065] Establish a composite loss function:

[0066] ;

[0067] In the above formula, Represents the main loss function. Represents the denoising loss function. and These are the weighting coefficients;

[0068] Main loss function The loss includes L1 loss, L2 loss, total variational loss, and structural similarity loss, as shown in the following formulas:

[0069] ;

[0070] In the above formula, , , and These are the weight coefficients for the four loss functions. The restored image. For a clear image, Indicates L1 loss, Indicates L2 loss, This represents the total variational loss. Represents structural similarity loss;

[0071] Denoising loss function This includes L1 loss, L2 loss, and total variational loss;

[0072] ;

[0073] In the above formula, , and These are the weight coefficients for the three loss functions. This indicates an 8x reduction in resolution. This represents the low-resolution output of the encoding network. For a clear image, For degenerate convolution kernels, It is a convolution operator.

[0074] In S3, the training process is as follows:

[0075] Using the composite loss function As a constraint, the restoration model is trained using degraded images from the training dataset. The encoder extracts features at different resolutions, and these features are then fed into a convolutional layer to supervise the denoising capabilities of the encoding module. The multi-level features extracted by the encoder are further deblurred by the decoder to generate the restored image. During training, the batch size is set to 1, and the existing Adam optimizer is used to update the parameters, corresponding to the first-order momentum. and second momentum The formula is as follows:

[0076] ;

[0077] ;

[0078] In the above formula, This is the first-order momentum from the previous step. Let be the second momentum of the previous moment. For the current gradient, The square of the current gradient. It is the first-order momentum coefficient, which controls the first-order momentum (exponential moving average of historical gradient) and determines the smoothness of the update direction (usually 0.9). The second-order momentum coefficient controls the second-order momentum (an exponential moving average of the squared historical gradients), determining the adaptive magnitude of the learning rate (typically 0.999). Set it to 0.9. The learning rate is set to 0.999, and the training process is divided into two stages: In the first stage, the learning rate is calculated using cosine annealing from... Upgraded to In the second stage, a cosine annealing strategy is used to reduce the learning rate from... Gradually decrease to .

[0079] A specific comparative experiment is provided below:

[0080] Images restored using the network provided by this invention are compared with images generated by common methods in the prior art on a simulation dataset. The comparison images include the original image after degradation, the deblurring algorithm EVSSm, the deblurring algorithm DyUnet, and the technical solution of this invention. The comparison metrics include Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), Mean Absolute Error (MAE), and Perceptual Loss (LPIPS). Regarding the comparison metrics, PSNR, SSIM, MAE, and LPIPS are reference image evaluation metrics used to evaluate the similarity between the restored image and the reference clear image. Higher PSNR and SSIM values ​​indicate a closer similarity between the restored image and the reference clear image, while lower MAE and LPIPS indicate a better image restoration effect. Figures 3-4 Each image in the table is accompanied by corresponding values ​​for PSNR, SSIM, MAE, and LPIPS in that order. A comparison of these four metrics shows that this invention has a stronger ability to restore images with different noise levels compared to other methods. Furthermore, its performance on real-world images demonstrates that the proposed method is closer to the characteristics of human visual perception. PSNR (Peak Signal-to-Noise Ratio) is improved by more than 0.25 dB, SSIM (Structural Similarity) by more than 0.0065, MAE by more than 0.003, and LPIPS by more than 0.0065. This indicates that the invention achieves improvements over other methods in all metrics and exhibits robust performance across a wide noise range.

[0081] exist Figure 5 In this paper, an evaluator is used to compare the images of the present invention with those processed by prior art. The evaluator's metrics include MUSIQ (Multi-Scale Quality Assessment Transformer), BRSIQUE (No Reference Spatial Domain Image Quality Evaluator), PI (Perception Index), NIQE (Natural Image Quality Evaluator), and ILNIQE (Intact Blind Image Quality Evaluator). MUSIQ, BRSIQUE, PI, NIQE, and ILNIQE are all no reference image evaluation metrics, reflecting the consistency between the restored image quality and human visual perception of image quality. A higher MUSIQ indicates that the image restoration effect is closer to human visual perception of image quality, while lower BRSIQUE, PI, NIQE, and ILNIQE indicate that the image restoration effect is closer to human visual perception of image quality. The metrics are labeled in the order of MUSIQ, BRSIQUE, PI, NIQE, and ILNIQE. Figure 5 Below each image, in Figure 5Compared with other technical solutions, the present invention improves MUSIQ by more than 1.40, reduces BRSIQUE by more than 4.6, reduces PI by more than 0.45, reduces NIQE by more than 0.19, and reduces ILNIQE by more than 2. Therefore, it can be seen that the present invention has achieved improvements in all indicators compared with other methods.

[0082] Therefore, this invention employs an image restoration method for a simple infrared lens imaging system with a wide signal-to-noise ratio range, providing a diffractive lens infrared image reconstruction network model and its reconstruction method that combine high reconstruction quality, strong noise generalization, and low computational complexity. This facilitates embedded deployment and promotes the practical application and lightweight development of diffractive lens infrared imaging technology.

[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for image restoration in a wide signal-to-noise ratio range infrared simple lens imaging system, characterized in that: Includes the following steps: S1: Collect a dataset of infrared simple lens imaging, and use clear and degraded images as training datasets for the restoration model; S2: Construct the Wavelet-ProxNet model as the restoration model. The restoration model adopts an asymmetric encoder-decoder structure, including an encoder and a decoder. The encoder includes an encoding module and a strided convolutional downsampling module, and the decoder includes a decoding module and an upsampling module. The encoder's encoding modules adopt the MetaFormer structure, including a region-biased convolutional attention module ABCAM and a wavelet transform-based feedforward module WaveletMLP. The region-biased convolutional attention module ABCAM consists of a 1×1 region-biased convolution, a 3×3 region-biased convolution, SimpleGate, a simple channel attention mechanism SCA, and a 1×1 region-biased convolution connected in sequence. The wavelet transform-based feedforward module WaveletMLP consists of a discrete wavelet transform module DWT, a 1×1 convolution, LeakyReLU, a 1×1 convolution, inverse discrete wavelet transform, a 3×3 convolution, LeakyReLU, and a 1×1 convolution connected in sequence. The decoding module includes a feature domain gradient descent module (GDM) and a proximal mapping module (PMM). S3: Establish the composite loss function As a constraint, the restoration model is trained using the training dataset; S4: For the image to be restored, the restored image is generated by the trained restoration model.

2. The image restoration method for a wide signal-to-noise ratio range infrared simple lens imaging system according to claim 1, characterized in that: In S1, the process of obtaining the training dataset is as follows: We collected high-quality open-source infrared datasets, performed degradation calibration on a simple infrared lens imaging system, convolved the calibrated point spread function (PSF) with the high-quality infrared dataset, added dynamic Gaussian noise to simulate detector noise, and obtained degraded images. We then combined the degraded images and the corresponding clear images to form the training dataset for the restoration model.

3. The image restoration method for a wide signal-to-noise ratio range infrared simple lens imaging system according to claim 1, characterized in that: In S2, the encoder's encoding module is as follows: The encoding module for input features Processed output as follows: ； Here, WaveletMLP represents a wavelet transform-based feedforward module. It is layer normalization.

4. The image restoration method for a wide signal-to-noise ratio range infrared simple lens imaging system according to claim 1, characterized in that: In S2, the encoder's strided convolution downsampling module is as follows: using strided convolution with a stride of 2, after the input features are processed by the strided convolution downsampling module, the feature space resolution is reduced to 1 / 2 of the input size, and the number of channels becomes twice that of the input.

5. The image restoration method for a wide signal-to-noise ratio range infrared simple lens imaging system according to claim 1, characterized in that: In S2, the decoding module in the decoder is as follows: The feature domain gradient descent module (GDM) uses residual blocks to predict the degradation matrix and its transpose. The iterative formula for gradient descent is obtained. as follows: ； In the above formula, It is the first Level decoder Sub-iteration decoder features, when hour For the first Level output Upsampling, Indicates the step size. This represents the predicted residual block of the degenerate matrix transpose. It is a degenerate matrix prediction residual block. It is the first Level encoder features; The Proximal Mapping Module (PMM) uses the NAFBlock module, and its output formula is as follows: ； In the above formula, CAM represents the convolutional attention module, LayerNorm represents layer normalization, and MLP represents the feedforward module. Indicates intermediate calculation quantities, set This is the upper limit of iterations, when the number of iterations... At that time, the gradient descent iterative formula and the output formula of the proximal mapping module are updated respectively. and .

6. The image restoration method for a wide signal-to-noise ratio range infrared simple lens imaging system according to claim 1, characterized in that: In S2, the upsampling module in the decoder consists of a subpixel convolution module and Conv2d, which are used to adjust the feature resolution and channels. After the input features are downsampled, the feature space resolution becomes twice the input size, and the number of channels becomes half of the input.

7. The image restoration method for a wide signal-to-noise ratio range infrared simple lens imaging system according to claim 1, characterized in that: In S3, the process of establishing the composite loss function is as follows: Establish a composite loss function: ； In the above formula, Represents the main loss function. Represents the denoising loss function. and These are the weighting coefficients; Main loss function The loss includes L1 loss, L2 loss, total variational loss, and structural similarity loss, as shown in the following formulas: ； In the above formula, , , and These are the weight coefficients for the four loss functions. The restored image. For a clear image, Indicates L1 loss, Indicates L2 loss, This represents the total variational loss. Represents structural similarity loss; Denoising loss function This includes L1 loss, L2 loss, and total variational loss; ； In the above formula, , and These are the weight coefficients for the three loss functions. This indicates an 8x reduction in resolution. This represents the low-resolution output of the encoding network. For a clear image, For degenerate convolution kernels, It is a convolution operator.

8. The image restoration method for a wide signal-to-noise ratio range infrared simple lens imaging system according to claim 1, characterized in that: In S3, the training process is as follows: Using the composite loss function As a constraint, the restoration model is trained using degraded images from the training dataset. The encoder extracts features at different resolutions, and these extracted features are fed into a convolutional layer to supervise the denoising capabilities of the encoding module. The multi-level features extracted by the encoder are then further deblurred by the decoder to generate the restored image. During training, the batch size is set to 1, and the Adam optimizer is used to update the parameters. Set it to 0.

9. Set it to 0.

999. This represents the first-order momentum coefficient. This represents the second-order momentum coefficient. The training process is divided into two stages: the first stage, the learning rate is calculated using cosine annealing from... Upgraded to ; In the second stage, a cosine annealing strategy is used to reduce the learning rate from... Gradually decrease to .

Images