Infrared super-resolution imaging method and system based on nerve compression self-coding

Abstract

The invention discloses an infrared super-resolution imaging method and an infrared super-resolution imaging system based on nerve compression self-coding, in a training stage, a physical imaging process is modeled into an optical degradation encoder according to a compression sensing theory, and a fuzzy operation is combined by introducing a learnable binarization mask, so that a super-resolution image is obtained. Simulating a sparse coding process of compressed sensing to obtain sparse representation of the input image; restoring the sparse representation into a high-resolution image by using a CNN-based super-resolution decoder; in the reasoning stage, the optical degradation encoder is replaced by a real sampling process, and the optimized binarization mask is used as a mask set in the spatial light modulator and is finally projected to an image collected by a sensor; and the trained CNN model is also stored and is used for carrying out a super-resolution task on a really captured coded image. According to the invention, an efficient and reliable solution is provided for super-resolution reconstruction of the low-resolution infrared image.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to an infrared super-resolution imaging method and system based on neural compressive auto-encoding. Background Art

[0002] In the fields of remote sensing, military reconnaissance, and medical imaging, there is a need for high-resolution images. However, due to factors such as the limitation of the pixel size of CCD sensors, the high cost of high-resolution CCD cameras, and the large volume and weight of these devices, it is impossible to meet the requirement of directly obtaining high-resolution images. Compressive sensing (CS) can simultaneously achieve signal sampling and compression, which helps to reduce the cost of sampling and transmission. However, traditional compressive sensing (CS) also has disadvantages. Although it has a small computational amount during image reconstruction, the quality of the reconstructed image is low, and it is difficult to recover the details and edges of the image well, which is also an urgent problem to be solved by CS. In addition, in order to obtain a better-quality reconstructed image, it is often necessary for the sensor to provide more relevant observation results, which increases the cost of the sensor and is not conducive to data compression.

[0003] In recent years, with the great progress made by deep learning methods, super-resolution models based on deep learning have been widely proposed, and generally can obtain better performance compared with various traditional super-resolution methods. Specifically, the convolutional neural network (CNN) method has achieved excellent performance in the SISR task. Therefore, CNN has been proposed to utilize the SISR task. The correlation between LR and HR images can be easily learned through deep learning-based techniques, and better performance can be obtained compared with traditional methods. In addition, these techniques can generate high-resolution images. However, existing algorithms still have some challenges and limitations. Traditional super-resolution methods, such as bicubic interpolation and sparse coding-based reconstruction, are limited by manually designed features and fixed sampling strategies, and it is difficult to adaptively capture the sparsity prior of the image, especially in the case of low-resolution input, the high-frequency details (such as edges and textures) are severely degraded. Existing research shows that the performance of interpolation algorithms dominated by low-frequency information significantly decreases in complex degradation scenarios. According to the compressive sensing theory, if a signal is sparse in a certain domain, the signal can be recovered through a sampling rate far lower than the Nyquist sampling theorem. An image can be regarded as a kind of signal, and the high-frequency information in the image is usually sparse, which means that the sparse representation of a high-resolution image can be obtained through a small number of random measurements (such as binary masks), and the original high-resolution image can be recovered through a reconstruction algorithm. Summary of the Invention

[0004] The purpose of the present invention is to provide an infrared super-resolution imaging method and system based on neural compressive auto-encoding.

[0005] To achieve the above object, the present invention provides the following technical solution: An infrared super-resolution imaging method based on neural compressive auto-encoding, comprising the following steps:

[0006] In the training stage, according to the compressive sensing theory, the physical imaging process is modeled as an optical degradation encoder. By introducing a learnable binary mask combined with a blurring operation, the sparse coding process of compressive sensing is simulated to obtain the sparse representation of the input image;

[0007] A super-resolution decoder based on CNN is used to restore the sparse representation into a high-resolution image; the optical degradation encoder and the super-resolution CNN decoder are jointly optimized during training, and the binary mask sequence in the optical degradation encoder and the parameters of the super-resolution CNN are updated simultaneously;

[0008] In the inference stage, the optical degradation encoder is replaced by the real sampling process, and the optimized binary mask is used as the mask set in the spatial light modulator, and finally projected onto the image collected by the sensor; the trained CNN model is saved for performing super-resolution tasks on the real captured encoded images.

[0009] Further, according to the compressive sensing theory, the signal sampling process can be represented by the following mathematical expression:

[0010]

[0011] In the formula is the true value of the original signal, is the measured value of the original signal, representing the sensed projection set, is the measurement matrix, defined by a set of random matrices, is the measurement noise;

[0012] The optical degradation encoder decomposes the measurement matrix in the compressive sensing imaging process into a learnable binary mask and a blurring kernel to achieve sparse sampling of the input image;

[0013]

[0014] Among them, represents matrix multiplication;

[0015] The learnable binary mask combines the sparse sampling theory of compressive sensing with the deep learning parameterization ability to achieve adaptive encoding of the input image;

[0016] The learnable binary mask dynamically optimizes the mask weight distribution through end-to-end training. The weights of the binary mask are initialized as trainable parameters, and during the network training process, these weights will be gradually updated according to the feedback of the loss function to optimize the quality of the reconstructed image;

[0017] The binary mask is a two-dimensional binary parameter tensor. The reparameterization method is adopted, that is, a learnable parameter vector , is an m-dimensional vector space in the real number field, and the required binary tensor is derived through the sign function :

[0018]

[0019]

[0020] Considering that the sign function is non-differentiable at zero and its derivative vanishes at other points, the reparameterization method introduces a clip function to replace the derivative of the sign function, so as to achieve backpropagation through gradient descent during the network training process; the clip function formula is as follows:

[0021]

[0022] The generation process of the binary mask is implemented through a feed-forward network. Specifically: the binarization function binarizes the mask matrix to generate a binary mask with 0 and 1 elements; the elements of the mask matrix are initialized as random values with a uniform distribution and are gradually adjusted during the training process to capture the key information regions in the input image; during the forward propagation process, the input image is multiplied element-wise with the generated mask to generate a masked feature map; this feature map retains the key information in the input image and suppresses the unimportant regions through the mask, thereby improving the processing efficiency and accuracy of the subsequent network modules.

[0023] Through the blur operation, the point spread function of the actual optical system is simulated, and the masked feature map is convolved with the blur kernel, which can smooth the image and mix information, enhance the sparsity of the signal and effectively reduce the high-frequency noise.

[0024] An infrared super-resolution imaging system based on neural compressive auto-encoding is used to implement the above method. The system includes:

[0025] The first module is used in the training stage to model the physical imaging process as an optical degradation encoder according to the compressive sensing theory. By introducing a learnable binary mask and combining the blur operation, the sparse coding process of compressive sensing is simulated to obtain the sparse representation of the input image;

[0026] The second module uses a CNN-based super-resolution decoder to restore the sparse representation into a high-resolution image; the optical degradation encoder and the CNN-based super-resolution decoder are jointly optimized during training; the binary mask sequence in the optical degradation encoder and the parameters of the super-resolution CNN are updated simultaneously;

[0027] The third module is used to implement: during the inference phase, the optical degradation encoder is replaced by the real sampling process, the optimized binary mask is used as the mask set in the spatial light modulator, and finally projected onto the image collected by the sensor; save the trained CNN model for performing super-resolution tasks on the real captured encoded images.

[0028] An electronic device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the above method are implemented.

[0029] A computer-readable storage medium stores a computer program thereon. When the program is executed by a processor, the steps of the above method are implemented.

[0030] A computer program product includes a computer program. When the computer program is executed by a processor, the steps of the above method are implemented.

[0031] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0032] (1) The present invention proposes an end-to-end collaborative optimization algorithm framework for infrared image super-resolution based on the compressed sensing theory and deep learning. By designing an optical degradation encoder and a super-resolution decoder module, and combining learnable masks and blur operations, efficient super-resolution reconstruction of infrared images is achieved; the optical degradation encoder simulates the sparse sampling process of compressed sensing, uses a learnable mask to enhance the high-frequency details of the image, and introduces information mixing through blur operations, improving the sparsity and noise resistance of the signal; the super-resolution decoder extracts features and reconstructs the sparse representation through a convolutional neural network, and finally generates high-quality high-resolution images.

[0033] (2) The framework proposed by the present invention shows high performance in terms of metrics (PSNR, SSIM), and can effectively restore the details and edge features of the image. The key roles of the mask and blur modules are further verified through ablation experiments. An appropriate number of masks and an appropriate blur kernel size can significantly improve the reconstruction effect.

[0034] (3) The end-to-end joint optimization mechanism designed by the present invention enables the optical degradation encoder and the decoder to work together, further improving the adaptability and reconstruction quality of the model.

[0035] (4)The present invention provides an efficient and reliable solution for the super-resolution reconstruction of low-resolution infrared images, which has important application values in the fields of low-light imaging, target recognition, remote sensing, etc. Description of the Drawings

[0036] Figure 1 It is a framework diagram of an infrared image super-resolution algorithm based on a learnable mask and a deep decoder.

[0037] Figure 2 It is a structural diagram of a super-resolution decoder based on learning.

[0038] Figure 3 It is a structural diagram of the Encoder.

[0039] Figure 4 It is a structural diagram of the Decoder.

[0040] Figure 5 It is a structural diagram of the Resblock.

[0041] Figure 6 They are partial simulated dataset images.

[0042] Figure 7 They are experimental effect diagrams of different structures.

[0043] Figure 8 They are effect diagrams of different numbers of masks. Detailed Implementation Manner

[0044] The present invention proposes an infrared super-resolution imaging method based on neural compressive auto-encoding. By jointly optimizing the sparse coding and deep learning reconstruction modules, efficient super-resolution of infrared images is achieved. Specifically, the binary mask operation is used to sparsely sample the input image in front of the image sensor, and the blur operation is used to simulate the blur effect of the image sensor. Through training, the binary mask parameters can be adaptively adjusted, so as to apply different sampling strategies in different regions of the image and capture the high-frequency detail information in the image in an optimal way. The super-resolution of the model is achieved in two main steps: first, the sparse representation of the image is obtained through the sparse coding module of compressive sensing, and then the sparse representation is restored to a high-resolution image through the deep learning decoder. In the sparse coding stage, the input image is processed by a learnable binary mask and blur to generate a low-resolution sparse measurement map. Then, the sparse map is encoded into a feature representation through the encoder. In the decoding stage, the decoder maps these features back to the high-resolution image space, thus achieving super-resolution reconstruction.

[0045] Compressed sensing is a signal acquisition paradigm that can sample signals at sub-Nyquist rates. Different from traditional sensing methods, in CS, only a small number of random linear measurement results are obtained instead of sensing the entire signal, and a reconstruction algorithm is used to recover the original signal from the measurement results. According to the theory of compressed sensing, the signal sampling process can be expressed by the following mathematical formula:

[0046]

[0047] In the formula is the true value of the original signal, is the measured value of the original signal, representing the sensed projection set, is the measurement matrix, defined by a set of random patterns (usually a random Gaussian matrix), is the measurement noise.

[0048] The overall process of the infrared image super-resolution algorithm framework based on a learnable mask and a deep decoder proposed in the present invention is as shown in Figure 1 In the training stage, according to the theory of compressed sensing, the physical imaging process is modeled as an optical degradation encoder. By introducing a learnable binary mask and combining with a blur operation, the sparse coding module of compressed sensing is simulated to obtain the sparse representation of the input image. Then, a super-resolution decoder based on CNN is used to restore the sparse representation into a high-resolution image. It should be noted that the optical degradation encoder and the super-resolution CNN decoder based on CNN are jointly optimized during training. In this way, the binary mask sequence in the optical degradation encoder and the parameters of the super-resolution CNN will be updated simultaneously, enabling us to find a solution better than separate optimization. In the inference stage, the optical degradation encoder is replaced by the real sampling process, and the optimized binary mask is used as the mask set in the spatial light modulator and finally projected onto the image captured by the sensor. The trained CNN model will also be saved for performing super-resolution tasks on the real captured coded images. The design and implementation of each module in the framework will be introduced in detail below.

[0049] The fixed observation matrix of traditional CS is difficult to adapt to complex scenarios. In the network framework of the present invention, a learnable degradation encoder is designed to simulate the optical degradation scenario faced in the real sensor sampling process; this encoder, through joint optimization of the binary mask and the blur operation, decomposes into a dynamic mask and a blur kernel to achieve sparse sampling of the input image.

[0050]

[0051] The present invention proposes a physically-guided Learnable Binary Mask (LBM) module to achieve the design of a learnable degradation encoder. The core lies in combining the sparse sampling theory of compressive sensing with the parameterization ability of deep learning to achieve adaptive encoding of input images. Different from traditional fixed-mask compressive sensing (CS), LBM dynamically optimizes the mask weight distribution through end-to-end training, with a focus on enhancing the sparse sampling efficiency for high-frequency details (such as edges and textures). The weights of the binary mask are initialized as trainable parameters, and during the network training process, these weights will be gradually updated according to the feedback of the loss function to optimize the quality of the reconstructed image. The binary mask designed in the present invention is a two-dimensional binary parameter tensor, and there are well-established solutions in deep learning to implement the backpropagation training of binary parameters. The present invention uses a widely used technique, namely the "straight-through estimator" (STE). Specifically, in the optical degradation encoder, instead of directly defining the binary tensor, we adopt a reparameterization trick, that is, introducing a learnable parameter vector , and deriving the required binary tensor through the sign function .

[0052]

[0053]

[0054] Considering that the sign function is non-differentiable at zero and its derivative vanishes (i.e., equals 0) at other points, the STE technique introduces a clip function to replace the derivative of the sign function, thereby achieving backpropagation through gradient descent during the network training process. The clip function formula is as follows:

[0055]

[0056] The generation process of the binary mask is implemented through a feed-forward network. The present invention designs a binary function to binarize the mask weight matrix to generate a binary mask with 0 and 1 elements. The elements of the mask matrix are initialized to random values with a uniform distribution and are gradually adjusted during the training process to capture the key information regions in the input image. During the forward propagation process, the input image is multiplied element-wise with the generated mask to generate a masked feature map. This feature map retains the key information in the input image while suppressing the unimportant regions through the mask, thereby improving the processing efficiency and accuracy of subsequent network modules. The present invention also introduces a blurring operation to simulate the point spread function of a spatial light modulator (SLM), which uses different Gaussian blur kernels to convolve the image, can smooth the image and mix information, enhance the sparsity of the signal, and effectively reduce high-frequency noise. In compressive sensing, the reconstruction of the signal depends on the design of the observation matrix, and in this work, this observation matrix is represented by a learnable mask matrix. Through the end-to-end training of the network, the mask matrix can be dynamically adjusted to meet the requirements of signal reconstruction. This ability of dynamic adjustment enables the network to better adapt to complex image features, thereby improving the accuracy during the reconstruction process. Through such a design, the network can effectively encode the key information of the input image while maintaining computational efficiency, ultimately enhancing the effect of super-resolution reconstruction.

[0057] The CNN-based super-resolution decoder (DSRE) mainly consists of two sub-modules, Encoder and Decoder, and its structural diagram is as Figure 2 shown. DSRE uses a deep convolutional neural network to gradually map the low-resolution feature map to the high-resolution space through layer-by-layer convolutional operations. The low-resolution image is encoded into a set of sparse maps after passing through the optical degradation encoder and then sent to the Encoder module for feature extraction. The extracted features are fused with the learnable Mask, and the fused features are further used to extract local feature information through a 3×3 convolutional layer; then Pixel Shuffle is used for upsampling to achieve the conversion from low-resolution to high-resolution image. Finally, the Decoder module further optimizes and reconstructs the high-resolution image through a series of convolutional layers. Encoder consists of 3 residual blocks (ResBlock) and 2 3×3 convolutional layers, and its structure is as Figure 3 shown. The structure of Dncoder is similar to that of Encoder, except that in the last layer, Decoder uses a 1×1 convolutional layer as the final output, and its structure is as Figure 4 shown. Compared with the traditional residual network structure, the residual blocks used in the present invention, such as Figure 5As shown, the BN layer (Batch Normalization) is removed. The main function of the BN layer is to normalize features, thereby reducing the network's dependence on initialization. However, the existence of the BN layer will increase the scale and complexity of the network to a certain extent. The core of image super-resolution reconstruction lies in the restoration of image details and textures, and there is not a high requirement for the correlation and consistency between features. At the same time, the BN layer also consumes a large amount of computing memory resources. Therefore, removing the BN layer can not only reduce the GPU memory occupancy but also enhance the ability to reconstruct image details.

[0058] In the field of image super-resolution, the loss function is used to measure the reconstruction error and guide model optimization. In the super-resolution decoder network based on CNN, the loss function is used to measure the reconstruction error and guide model optimization. There are mainly the following three groups of loss functions;

[0059] The first group of Loss is the multi-scale Charbonnier loss, which is used to penalize the deviation between the estimated sharp image and its real image at different spatial scales, as shown in Equation (7):

[0060]

[0061] where N is the total number of samples; 、 and represent the number of pixels, the estimated sharp image, and the corresponding real image at scale n respectively, is a small constant to ensure differentiability, and is set to according to experience.

[0062] The second group of Loss is the multi-scale edge loss, defined as shown in Equation (8):

[0063]

[0064] where represents the Laplacian operator, which constrains the edges of the restored sharp image to be highly consistent with the edges of the real image.

[0065] The third group of Loss is the structural similarity (SSIM) loss, which is used to constrain the differences between the restored sharp image and the real image in terms of structure, brightness, and contrast, as shown in Equation (9):

[0066]

[0067] where N is the number of windows, is the SSIM value of the th window, and the SSIM calculation method is as shown in Equation (10):

[0068]

[0069] wherein , are the local average brightness of image x and image y respectively, , are the local variances of image x and image y respectively, is the covariance of image x and image y; , is a constant to avoid a zero denominator, and can be set to , , where L is the dynamic range of the image pixel values, taking 255, , is a small constant, taking 0.01 and 0.03.

[0070] In summary, the final loss function is shown in formula (11):

[0071]

[0072] wherein , are weight coefficients. Exemplarily, , are both set to 1.

[0073] Based on the same inventive concept, the present invention also proposes an infrared super-resolution imaging system based on neural compressive auto-encoding for implementing the above method. The system includes:

[0074] The first module is used to, in the training stage, model the physical imaging process as an optical degradation encoder according to the compressive sensing theory, and simulate the sparse coding process of compressive sensing by introducing a learnable binary mask in combination with a blurring operation to obtain a sparse representation of the input image;

[0075] The second module uses a CNN-based super-resolution decoder to restore the sparse representation into a high-resolution image; the optical degradation encoder and the CNN-based super-resolution decoder are jointly optimized during training; the binary mask sequence in the optical degradation encoder and the parameters of the super-resolution CNN are updated simultaneously;

[0076] The third module is used to implement: in the inference stage, the optical degradation encoder is replaced by the real sampling process, the optimized binary mask is used as the mask set in the spatial light modulator, and finally projected onto the image collected by the sensor; save the trained CNN model for performing the super-resolution task on the real captured encoded image.

[0077] The specific implementation methods of the above first module to third module are the same as the steps of the aforementioned infrared super-resolution imaging method based on neural compressive auto-encoding, and the present invention will not elaborate further.

[0078] The present invention will be described in detail below in conjunction with embodiments and the accompanying drawings.

[0079] Embodiment

[0080] In this embodiment, a simulated dataset will be used for ablation and comparison experiments to verify the effectiveness and reliability of the proposed method through experiments, and a built real optical system will be used to collect real infrared images to verify the image super-resolution effect of the proposed algorithm framework.

[0081] The simulated dataset Drek Vision Dataset (DVD) is used for high-quality low-light imaging tasks of RGB image and NIR (near-infrared image) fusion. In the article, all RGB-NIR image pairs are collected by switching the optical filters directly placed in front of the camera, and they are divided into two types of image pairs for different purposes. The present invention uses its NIR images of 2672×1504 collected in low-light environments as the simulated dataset, and adds Gaussian noise with a standard deviation range of 0 to 0.02 to simulate the physical imaging process. This dataset meets the requirements of the tasks solved by the present invention, and the NIR super-resolution reconstruction under low light can better reflect the effectiveness of the algorithm framework proposed by the present invention. In the following experiments, 1500 images with a resolution of (890×752) are used as the training set, and 100 images with a resolution of (890×752) are used as the test set. Some images of the simulated dataset are as Figure 6 shown. The images in the training set and the test set do not contain each other.

[0082] The present invention is trained in an end-to-end manner on the Pytorch framework; all experiments are carried out on NVIDIA GeForce RTX 4070. In the training stage, the Adam optimizer is used to update the parameters and the learning rate is initialized to , the number of samples selected for one training, Batch size, is set to 8, and the number of training epochs for all training samples is 200; for each training iteration, the image is randomly cropped to a size of 256×256, and random flipping and rotation operations are used to enhance the diversity of the data.

[0083] To verify the effectiveness of the proposed end-to-end collaborative optimization framework for infrared image super-resolution based on compressive sensing theory and deep learning, a series of ablation experiments will be carried out in this embodiment for verification, mainly including structures such as the super-resolution decoder module, the mask in image preprocessing, and the blur kernel. And two metrics, PSNR and SSIM, are used for quantitative evaluation.

[0084] Structure 1 only uses the super-resolution decoder module for image super-resolution reconstruction without adding masking and blurring operations, verifies the image super-resolution ability of the proposed super-resolution decoder module, and conducts ablation comparison with other modules to demonstrate the effectiveness of the proposed module.

[0085] Structure 2 uses different sizes of blur kernels on the basis of the proposed network framework (Structure 1) to analyze the influence of different blur kernels on this network framework.

[0086] Structure 3 uses different numbers of masks on the proposed network framework (Structure 1) to analyze the influence of different numbers of masks on this network framework.

[0087] Table 1 Quantitative evaluation results of different structures

[0088]

[0089] To verify the influence of the proposed masking and blurring modules on the image super-resolution task of the present invention, the present invention designs ablation experiments for four modules. Structure 1 only uses the decoder module, Structure 2 uses a blur kernel with a Gaussian blur kernel K of 5 for blurring on the basis of the decoder module, the number of masks N in Structure 3 is 10, and the structure of the present invention is to add blurring and masking operations with K = 5 and N = 10 on the basis of Structure 1. To explore the influence of different specific numbers of masks and blur kernels on the proposed network framework, the present invention designs different settings of N = 1, N = 2, and N = 10 when the blur kernel K = 5 for effect verification, and at the same time sets different blur kernel sizes when N = 10 for experimental analysis. After the above experiments converge, the low-resolution images in the test set are respectively input, and the PSNR and SSIM metrics are calculated for the super-resolution images output by the model and the original high-resolution images. Tables 1, 2, and 3 show the quantitative evaluation results of different structures, which are used to objectively measure the super-resolution effects of different structures. At the same time Figure 7 , Figure 8 shows the visualization diagrams of local details for subjective visual evaluation. Combining Table 1 and Figure 7 it can be clearly seen that the masking and blurring modules have obvious improvements in terms of indicators and visually on the image super-resolution ability of the proposed network framework. Compared with only using Structure 2 or Structure 3, the structure of the present invention can well reconstruct the edge parts of image details and gray-level mutations, strongly verifying the effectiveness of the scheme. Combining Table 2 and Figure 8It can be seen that as the number of masks increases, the indicators improve and the visualization effect becomes gradually obvious. However, the number of masks is not the more the better. Excessive masks will lead to too much information redundancy and an increase in computing resources. In this experiment, an ideal reconstruction effect can be achieved under the setting of N = 10. Table 3 shows the influence of different blur kernels. It can be seen that with different values of K, the evaluation indicators do not change much.

[0090] Table 2 Quantitative evaluation results of different masks

[0091]

[0092] Table 3 Quantitative evaluation results of different blur kernels

[0093]

[0094] The present invention proposes an end-to-end collaborative optimization algorithm framework for infrared image super-resolution based on the theory of compressive sensing and deep learning. By designing an optical degradation encoder and a super-resolution decoder module, and combining learnable masks and blur operations, efficient super-resolution reconstruction of infrared images is achieved. The optical degradation encoder simulates the sparse sampling process of compressive sensing, uses learnable masks to enhance the high-frequency details of the image, and introduces information mixing through blur operations, improving the sparsity and noise resistance of the signal. The super-resolution decoder then extracts features and reconstructs the sparse representation through a convolutional neural network, finally generating high-quality high-resolution images.

[0095] Experimental results show that the method proposed in the present invention exhibits high performance in terms of indicators (PSNR, SSIM), and can effectively restore the details and edge features of the image. The ablation experiment further verifies the key role of the mask and blur modules. An appropriate number of masks and an appropriate blur kernel size can significantly improve the reconstruction effect. At the same time, the end-to-end joint optimization mechanism designed in the present invention enables the optical degradation encoder and the decoder to work together, further improving the adaptability and reconstruction quality of the model. In summary, the method of the present invention provides an efficient and reliable solution for the super-resolution reconstruction of low-resolution infrared images, and has important application value in the fields of low-light imaging, target recognition, and remote sensing.

Claims

1. An infrared super-resolution imaging method based on neural compressive auto-encoding, characterized in that, It includes the following steps: In the training stage, according to the compressive sensing theory, the physical imaging process is modeled as an optical degradation encoder. By introducing a learnable binary mask combined with a blurring operation, the sparse coding process of compressive sensing is simulated to obtain the sparse representation of the input image; A CNN-based super-resolution decoder is used to restore the sparse representation to a high-resolution image; the optical degradation encoder and the CNN-based super-resolution decoder are jointly optimized during training, and the binary mask sequence in the optical degradation encoder and the parameters of the super-resolution CNN are updated simultaneously; In the inference stage, the optical degradation encoder is replaced by the actual sampling process, and the optimized binary mask is used as the mask set in the spatial light modulator and finally projected onto the image captured by the sensor; the trained CNN model is saved for performing super-resolution tasks on the actually captured coded images.

2. The infrared super-resolution imaging method based on neural compressive auto-encoding according to claim 1, characterized in that According to the compressive sensing theory, the signal sampling process is represented by the following mathematical expression: ； where is the true value of the original signal, is the measured value of the original signal, representing the sensed projection set, is the measurement matrix, defined by a set of random matrices, is the measurement noise; The optical degradation encoder jointly optimizes the binary mask and the blurring operation to decompose the measurement matrix in the compressive sensing imaging process into a learnable binary mask and the blurring kernel to achieve sparse sampling of the input image; ； Among them, represents matrix multiplication; The learnable binary mask combines the sparse sampling theory of compressive sensing with the parameterization ability of deep learning to achieve the adaptive coding of the input image; the learnable binary mask dynamically optimizes the mask weight distribution through end-to-end training. The weights of the binary mask are initialized as trainable parameters, and during the network training process, these weights will be gradually updated according to the feedback of the loss function to optimize the quality of the reconstructed image.

3. The infrared super-resolution imaging method based on neural compressive auto-encoding according to claim 2, wherein The binary mask is a two-dimensional binary parameter tensor, and a reparameterization method is adopted, that is, a learnable parameter vector is introduced , is an m-dimensional vector space over the real number field, and the required binary tensor is derived through the sign function : ； ； The reparameterization method introduces a clip function to replace the derivative of the sign function, so as to achieve backpropagation through gradient descent during the network training process; the formula of the clip function is as follows: ； The generation process of the binary mask is implemented through a feedforward network. Specifically, the binary function binarizes the mask matrix to generate a binary mask with 0 and 1 elements; the elements of the mask matrix are initialized as random values with a uniform distribution and are gradually adjusted during the training process to capture the key information regions in the input image; During the forward propagation process, the input image is multiplied element by element with the generated mask to generate a masked feature map; Through a blurring operation, the point spread function of the actual optical system is simulated, and the blurred kernel is used to perform convolution processing on the masked feature map to smooth the image and mix the information.

4. The infrared super-resolution imaging method based on neural compressive auto-encoding according to claim 1, wherein A CNN-based super-resolution decoder is used to restore the sparse representation to a high-resolution image; during training, the optical degradation encoder and the CNN-based super-resolution decoder are jointly optimized, and the binary mask sequence in the optical degradation encoder and the parameters of the super-resolution CNN will be updated simultaneously; Specifically; The CNN-based super-resolution decoder consists of two sub-modules, namely Encoder and Decoder. Utilizing a deep convolutional neural network, through layer-by-layer convolutional operations, it gradually maps the low-resolution feature map to the high-resolution space. The low-resolution image is encoded into a set of sparse maps after passing through the optical degradation encoder, and then fed into the Encoder module for feature extraction. The extracted features are fused with a learnable Mask, and the fused features are further used to extract local feature information through a 3×3 convolutional layer. Then, Pixel Shuffle is used for upsampling to achieve the conversion from low resolution to high resolution. Finally, the Decoder module further optimizes and reconstructs the high-resolution image through convolutional layers. The Encoder module consists of 3 residual blocks and 2 3×3 convolutional layers, and the Dncoder module uses a 1×1 convolutional layer as the final output. The BN layer is removed from the residual blocks used in both the Encoder module and the Decoder module.

5. The infrared super-resolution imaging method based on neural compressive auto-encoding according to claim 1, characterized in that In the CNN-based super-resolution decoder network, the loss function is used to measure the reconstruction error and guide the model optimization, mainly including the following three groups of loss functions; The first set of Loss is the multi-scale Charbonnier loss, which is used to penalize the deviation between the estimated clear image and its ground truth image at different spatial scales. The multi-scale Charbonnier loss is shown as follows: ； where N is the total number of samples; , and represent the number of pixels at scale n, the estimated clear image, and the corresponding ground truth image, respectively, is a constant; The second set of Loss is the multi-scale edge loss , which is defined as follows: ； wherein represents the Laplace operator; The third group of Loss is the structural similarity loss , which is used to constrain the differences between the restored clear image and the real image in terms of structure, brightness, and contrast, as shown in the following formula: ； where N is the number of windows, is the SSIM value of the th window, and the SSIM calculation method is as follows: ； where , are the local average brightness of image x and image y respectively, , are the local variances of image x and image y respectively, is the covariance of image x and image y; , is a constant to avoid a zero denominator; The final loss function is as follows: ； Among them and are weighting coefficients.

6. The infrared super-resolution imaging method based on neural compressive auto-encoding according to claim 5, characterized in that, , The values are all 1.

7. An infrared super-resolution imaging system based on neural compressive auto-encoding, characterized in that, To implement the method according to any one of claims 1 to 6, the system includes: The first module is used to model the physical imaging process as an optical degradation encoder according to the compressive sensing theory during the training phase. By introducing a learnable binary mask and combining with a blurring operation, it simulates the sparse coding process of compressive sensing to obtain the sparse representation of the input image. The second module uses a CNN-based super-resolution decoder to restore the sparse representation into a high-resolution image. The optical degradation encoder and the CNN-based super-resolution decoder are jointly optimized during training. The binary mask sequence in the optical degradation encoder and the parameters of the super-resolution CNN will be updated simultaneously. The third module is used to implement: during the inference phase, the optical degradation encoder is replaced by the real sampling process, and the optimized binary mask is used as the mask set in the spatial light modulator and finally projected onto the image collected by the sensor. Save the trained CNN model for performing the super-resolution task on the real captured encoded image.

8. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method according to any one of claims 1-6.

10. A computer program product comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1-6.

Images