An Infrared Image Super-Resolution Reconstruction Method Based on Channel Separation Deep Neural Network

Abstract

The present invention discloses an infrared image super-resolution reconstruction method based on channel separation deep neural network, which includes the following steps: Prepare a training data set: According to the existing high-resolution infrared image data set, simulate the degradation process through bicubic interpolation to generate corresponding low-resolution infrared images; Construct a deep neural network model: including a shallow feature extraction module, a deep feature extraction module, and an image reconstruction module; Train the deep neural network model: Set the hyperparameters, input the paired low-resolution and high-resolution infrared images into the network model, and update the network weights through the backpropagation algorithm until the set maximum number of iterations is reached; Save the network model for subsequent use in testing the infrared image super-resolution reconstruction effect. The method can improve the resolution of the captured low-resolution infrared images while better restoring their edges and textures, providing a solid foundation for subsequent motion segmentation and tracking, etc.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing methods, and particularly relates to an infrared image super-resolution reconstruction method based on channel separation deep neural network. Background Art

[0002] The visible spectral range that can be perceived by the human visual system is approximately 380nm to 720nm. Based on the principle of human vision, RGB cameras receive visible light waves reflected by objects in the scene to capture scene images. However, in scenarios such as at night or in bad weather, the limited light makes it almost impossible for the images obtained by conventional RGB cameras to retain details, seriously affecting the visual experience and information acquisition. In the above scenarios, external lighting, such as fill lights, can be used to increase the reflected light of the captured object to a certain extent, but this correspondingly leads to an increase in cost. Under extreme visible conditions, in order to image objects beyond the human perception ability, thermal imaging technology can be used, which can visualize the scene even at night or in the presence of fog or smoke. An infrared camera is a passive sensor that can sense the infrared radiation emitted by all objects with a temperature higher than absolute zero. Therefore, the imaging quality of infrared cameras is hardly affected under complex conditions such as insufficient light or bad weather conditions and does not require any external light source. In recent years, the progress of infrared imaging technology has enabled it to be extremely widely applied in fields such as military, medical, pedestrian detection, and maritime, with great application prospects.

[0003] Although infrared cameras can capture images in extremely challenging scenarios, due to the signal-to-noise ratio limitation in the sensor area of infrared cameras, the spatial resolution of infrared sensors cannot exceed a certain range. Therefore, compared with RGB cameras that usually provide a resolution of millions of pixels, the spatial resolution of general civilian infrared cameras is often low. On the other hand, simply increasing the size of the infrared sensor makes the production cost unaffordable. In addition, in order to achieve accurate thermal radiation measurement, infrared sensors are usually packaged in a separate vacuum package, and this process is time-consuming.

[0004] In order to solve the hardware limitations of infrared sensors and improve the resolution of infrared images, a method called image super-resolution (SR) has emerged. Single Image Super-Resolution (SISR) is to restore the corresponding high-resolution version of a single degraded low-resolution image based on the observed image. In recent years, with the improvement of integrated circuit design and manufacturing technology, the use of large-capacity GPUs with high-speed computing has become more popular, so infrared image super-resolution algorithms based on deep neural networks have made great progress. Deep neural networks trained with GPUs based on massive data are significantly better than traditional super-resolution methods based on interpolation and sparse representation in performance. However, most of the current infrared image super-resolution algorithms based on deep neural networks have a large number of redundant features that are constantly calculated due to the deep number of network layers, which increases the burden of network training, increases the difficulty of deployment, and limits their practical application value. In summary, the study of efficient infrared image super-resolution algorithms based on deep neural networks has clear practical significance and practical application value. Summary of the invention

[0005] The technical problem to be solved by the present invention is how to provide an infrared image super-resolution reconstruction method that can improve the resolution of a captured low-resolution infrared image while better restoring its edge and texture, thereby providing a solid foundation for subsequent motion segmentation and tracking.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is: a method for super-resolution reconstruction of infrared images based on a channel separation deep neural network, characterized in that it includes the following steps:

[0007] S1) preparing a training data set: based on an existing high-resolution infrared image data set, simulating the degradation process through bicubic interpolation to generate a corresponding low-resolution infrared image;

[0008] S2) constructing a deep neural network model: including a shallow feature extraction module, a deep feature extraction module and an image reconstruction module;

[0009] S3) training the deep neural network model: setting hyperparameters, inputting pairs of low-resolution and high-resolution infrared images into the network model, and updating the network weights through the back propagation algorithm until the set maximum number of iterations is reached;

[0010] S4) Save the network model for subsequent testing of infrared image super-resolution reconstruction effect.

[0011] A further technical solution is that the specific method of preparing the training data set includes:

[0012] For the collected public infrared image dataset, the bicubic interpolation downsampling algorithm is used to simulate the image degradation process, and the low-resolution infrared images corresponding to the scale factors of s = 2, s = 3, and s = 4 are obtained respectively. The above process can be expressed as:

[0013]

[0014] Among them, I HR represents the existing high-resolution infrared image, and f bicubic (·) represents the bicubic interpolation downsampling operation, is the corresponding low-resolution infrared image obtained by downsampling at the scale factor of s.

[0015] Furthermore, the shallow feature extraction module consists of two convolutional layers with a convolutional kernel size of 3×3, and their number of channels is 20 and 40 in sequence; the shallow feature extraction module is used to initially extract shallow features from the input low-resolution infrared image. The operations within the shallow feature extraction module can be expressed as:

[0016]

[0017] Among them, f i×i (·) represents the convolutional operation with a convolutional kernel size of i×i, and I SFE represents the output of the shallow feature extraction module.

[0018] Furthermore, the deep feature extraction module includes 16 channel-separated pixel attention residual modules for extracting high-frequency features; the channel separation method is used, that is, the output of each channel-separated pixel attention residual module is first separated in the channel dimension. The features of a part of the channels are sent to the next channel-separated pixel attention residual module to continue extracting features, and a part passes through the bypass feature extraction module. The channel separation operation can be expressed as:

[0019]

[0020] Among them, f channel-split (·) represents the channel separation operation, x i represents the output of the i-th channel-separated pixel attention residual module; represents the features sent to the next CS-PA block for processing after channel separation, represents the features sent to the bypass feature extraction module for processing; in particular:

[0021]

[0022] Among them, represents the input of the first channel-separated pixel attention residual module, Without bypass feature processing, directly connect the output of the pixel attention residual module separated from the last channel in the channel dimension;

[0023] The bypass feature processing module simply and efficiently realizes feature extraction by connecting bypass features in the channel dimension. The operations within the bypass feature processing module can be expressed as:

[0024]

[0025] Among them, I Bypass represents the total output of the bypass feature extraction module, and f c (,) represents connecting the feature maps along the channel dimension;

[0026] The operations within the 16-channel separated pixel attention residual module can be expressed as:

[0027]

[0028] Among them, I Trunk represents the output of the 16th channel separated pixel attention residual module;

[0029] After the input of the channel separated pixel attention residual module is separated by channels, a part of the features is processed by the pixel attention module, and the other part of the features passes through a convolutional layer with only one convolutional kernel of size 3×3, and residual connections are used inside the channel separated pixel attention residual module.

[0030] The total output of the deep feature extraction module can be expressed as:

[0031]

[0032] A further technical solution is that the image reconstruction module includes a feature reconstruction module and a convolutional layer with a convolutional kernel of size 1×1. The operations within the image reconstruction module can be expressed as:

[0033] I IR = f 1×1 (f FR (I DFE ))

[0034] Among them, f FR (·) represents the operations inside the image reconstruction module;

[0035] So far, the final output of the network can be expressed as:

[0036] I SR = f nearest (I LR ) + I IR

[0037] Among them, f nearest(·) represents nearest neighbor interpolation, I SR represents the finally reconstructed high-resolution infrared image.

[0038] A further technical solution lies in that the specific method for training the constructed deep neural network model includes:

[0039] Determine the optimizer, learning rate, and maximum number of iterations used for training, and start training the network; after each forward propagation is completed, calculate the L1 loss between I SR and I LR Then continue to update the network parameters through the backpropagation algorithm; repeat the above steps until the preset maximum number of iterations is reached, thus completing the network training.

[0040] A further technical solution lies in that the specific method for saving the model includes: solidifying a set of network weights corresponding to the highest evaluation index during the evaluation process of network training, and saving them as the final network model for later testing and reconstructing high-resolution infrared images.

[0041] The beneficial effects produced by adopting the above technical solutions are as follows:

[0042] The method of the present invention uses channel separation technology to reduce redundant feature extraction. Compared with the current mainstream deep neural network for infrared image super-resolution reconstruction that uses the same number of channel feature maps in each convolutional layer, the channel separation technology proposed by the present invention separates the generated features Figure 1 Part of them extracts deep features through the backbone deep feature extraction module, and part of the separated features pass through bypass features with fewer parameters, effectively reducing network parameters and redundant calculations.

[0043] The key component of the backbone deep feature extraction module of the method of the present invention, the channel separation pixel attention residual module, also uses the proposed channel separation technology to further reduce redundant calculations, efficiently extracts features through the pixel attention branch block and the non-attention branch block, and improves training stability through residual connections.

[0044] The infrared image super-resolution reconstruction method based on channel separation deep neural network proposed by the present invention essentially belongs to a lightweight model, has a small number of parameters, has the potential to be deployed on embedded devices and edge devices, and has good infrared image super-resolution reconstruction effects. Description of the Drawings

[0045] The present invention will be further described in detail below with reference to the drawings and specific embodiments.

[0046] Figure 1 is a flowchart of a preferred embodiment of the method of the present invention;

[0047] Figure 2 It is the network structure diagram of a preferred embodiment of the method of the present invention;

[0048] Figure 3 It is the structure diagram of the channel-separated pixel attention residual module of a preferred embodiment of the method of the present invention;

[0049] Figure 4 It is the structure diagram of the feature reconstruction block of a preferred embodiment of the method of the present invention;

[0050] Figure 5 It is the comparison chart of the ×2 scale factor super-resolution results of a preferred embodiment of the method of the present invention and a comparative method on two infrared image test datasets with different resolutions;

[0051] Figure 6 It is the comparison chart of the ×3 scale factor super-resolution results of a preferred embodiment of the method of the present invention and a comparative method on two infrared image test datasets with different resolutions;

[0052] Figure 7 It is the comparison chart of the ×4 scale factor super-resolution results of a preferred embodiment of the method of the present invention and a comparative method on two infrared image test datasets with different resolutions. Detailed implementation manners

[0053] Next, in conjunction with the accompanying drawings in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0054] In the following description, many specific details are set forth in order to fully understand the present invention, but the present invention can also be implemented in other ways different from those described herein. Those skilled in the art can make similar generalizations without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below.

[0055] As Figure 1 shown, the embodiments of the present invention disclose an infrared image super-resolution reconstruction method based on a channel-separated deep neural network, including the following steps:

[0056] S1) Prepare the training dataset: Simulate the degradation process of the collected public infrared image dataset through the bicubic interpolation downsampling algorithm to obtain the low-resolution infrared images corresponding to the scale factors s = 2, s = 3, and s = 4 respectively. The above process can be expressed as:

[0057]

[0058] Among them, I HR represents the existing high-resolution infrared image, and f bicubic (·) represents bicubic interpolation downsampling, which is the corresponding low-resolution infrared image obtained by downsampling with the scale factor s.

[0059] S2) Construct a deep neural network model: The constructed channel separation deep neural network mainly consists of the following three parts: a shallow feature extraction (Shallow Feature Extraction, SFE) module, a deep feature extraction (Deep Feature Extraction, DFE) module, and an image reconstruction (Image Reconstruction, IR) module. Among them, the shallow feature extraction module includes two convolutional layers with a convolutional kernel size of 3×3, and the number of channels is 20 and 40 in sequence. The SFE module is used to initially extract shallow features from the input low-resolution infrared image. The operations within the SFE module can be expressed as:

[0060] Among them, f i×i (·) represents a convolutional operation with a convolutional kernel size of i×i, and I SFE represents the output of the SFE module.

[0061] For I SFE using the channel separation technology, a part of the partial feature maps pass through the DFE module composed of 16 channel split pixel attention (Channel Split Pixel Attention, CS-PA) residual modules to extract high-frequency features. The structural diagram of the DFE module is as shown in Figure 2 . Since the number of layers of the deep neural network is relatively large, a large number of similar features are extracted by multiple layers. Therefore, a large number of redundant features are repeatedly extracted, resulting in a complex network structure and a large number of parameters, which is not conducive to deployment on embedded and edge devices. To solve the above problems, the present invention proposes a channel separation method, that is, the output of each CS-PA block is first separated in the channel dimension. The features of a part of the channels are sent to the next CS-PA block to extract features, and a part of the features are extracted through a bypass feature extraction module with a relatively simple structure. This channel separation method effectively reduces the redundant feature calculation and reduces the number of network parameters, enabling it to be efficiently deployed on embedded and edge devices. The channel separation operation can be expressed as:

[0062]

[0063] Among them, f channel-split (·) represents the channel separation operation, and x i represents the output of the i-th CS-PA block. Represents the feature sent to the next CS-PA block for processing after channel separation. Represents the feature sent for bypass processing. Specifically:

[0064]

[0065] Among them, Represents the input of the first CS-PA block, Without passing through the bypass feature processing, it is directly connected to the output of the last CS-PA block in the channel dimension.

[0066] To reduce the number of parameters, the bypass feature processing module proposed in the present invention simply and efficiently realizes feature extraction by connecting bypass features in the channel dimension, and its structure is as Figure 2 shown. The operations within the bypass feature processing module can be expressed as:

[0067]

[0068] Among them, I Bypass Represents the total output of the bypass feature extraction module, f c (,) represents the connection in the channel dimension.

[0069] The operations within 16 CS-PA blocks can be expressed as:

[0070]

[0071] Among them, I Trunk Represents the output of the 16th CS-PA block.

[0072] To minimize redundant feature calculations and improve the efficiency of infrared image super-resolution reconstruction, the feature channel separation technology is still used within the CS-PA block designed in the present invention, and the CS-PA block structure diagram is as Figure 3 shown. After the input of the CS-PA block passes through channel separation, a part of the features is sent to the Pixel Attention (PA) block for processing, and its structure diagram is as Figure 3 shown. Another part of the features passes through a convolutional layer with only one convolutional kernel of size 3×3. To improve the training stability, the present invention uses residual connections within the CS-PA block.

[0073] In summary, the total output of the DFE module can be expressed as:

[0074]

[0075] In the above method, the IR module is mainly composed of a Feature Reconstruction (FR) block and a convolutional layer with a convolutional kernel size of 1×1, and the structure diagram of the FR module is asFigure 4 As shown. The operations within the IR module can be expressed as:

[0076] I IR = f 1×1 (f FR (I DFE ))

[0077] where f FR (·) represents the internal operation of the FR module.

[0078] Up to this point, the final output of the network can be expressed as:

[0079] I SR = f nearest (I LR ) + I IR

[0080] where f nearest (·) represents nearest neighbor interpolation, and I SR represents the finally reconstructed high-resolution infrared image.

[0081] 3) Training the deep neural network: Determine the settings such as the optimizer, learning rate, maximum number of iterations, etc. used for training, and start training the network. Calculate the L1 loss between I SR and I LR , and update the network parameters through the backpropagation algorithm. Repeat the above steps until the pre-set maximum number of iterations is reached to complete the network training.

[0082] 4) Saving the model: Fix the set of network weights corresponding to the highest evaluation metric during the evaluation process and save it as the final network model for later testing of reconstructing high-resolution infrared images.

[0083] The present invention downsamples the infrared image test datasets with resolutions of 640×480 and 384×288 respectively using the Bicubic algorithm to obtain the corresponding low-resolution images, and then uses Bicubic, FSRCNN, SR-LUT, and the method of the present invention to perform super-resolution reconstruction on the low-resolution images. The super-resolution reconstruction results at ×2, ×3, and ×4 scale factors are respectively as Figure 5 , Figure 6 and Figure 7 shown. From Figure 5 , Figure 6 and Figure 7 it can be seen that at different scale factors, the infrared image super-resolution reconstruction results of the method of the present invention have richer edges and textures and fewer artifacts compared with the comparison algorithms.

[0084] The average PSNR and SSIM metrics of the method of the present invention, compared with the Bicubic, FSRCNN, and SR-LUT comparison methods, on the infrared image test datasets with resolutions of 640×480 (a total of 50 images) and 384×288 (a total of 20 images) are shown in Table 1. As can be seen from Table 1, the method of the present invention is superior to the comparison methods in both PSNR and SSIM quantitative metrics, demonstrating the effectiveness of the method of the present invention.

[0085] Table 1 Quantitative comparison results of infrared image super-resolution reconstruction between the present invention and the comparison methods

[0086]

[0087]

[0088] In addition, the model proposed by the method of the present invention is essentially a lightweight model, with a total of about 141K parameters, which is far lower than the mainstream infrared image super-resolution models. From the qualitative and quantitative comparisons, it can be seen that the method of the present invention can still achieve high quantitative metrics while maintaining a small number of parameters, and can better restore high-frequency information such as details and edges, and has the potential to be applied to embedded devices and edge devices.

Claims

1. An infrared image super-resolution reconstruction method based on a channel-separated deep neural network, characterized in that The steps include: S1) preparing a training data set: based on an existing high-resolution infrared image data set, simulating the degradation process through bicubic interpolation to generate a corresponding low-resolution infrared image; S2) constructing a deep neural network model: including a shallow feature extraction module, a deep feature extraction module and an image reconstruction module; S3) training the deep neural network model: setting hyperparameters, inputting pairs of low-resolution and high-resolution infrared images into the network model, and updating the network weights through the back propagation algorithm until the set maximum number of iterations is reached; S4) saving the network model for subsequent testing of infrared image super-resolution reconstruction effect; The shallow feature extraction module consists of two convolution layers with convolution kernel sizes of 3×3 and the number of channels is 20 and 40 respectively. The shallow feature extraction module is used to preliminarily extract shallow features from the input low-resolution infrared image. The operations in the shallow feature extraction module can be expressed as: Among them, f i×i (·) represents a convolution operation with a convolution kernel size of i×i, and I SFE represents the output of the shallow feature extraction module; The deep feature extraction module includes 16 channel separation pixel attention residual modules for extracting high-frequency features. The channel separation method is used, that is, the output of each channel separation pixel attention residual module is first separated in the channel dimension, and the features of part of the channel are sent to the next channel separation pixel attention residual module to continue extracting features, and part of the features are sent to the bypass feature extraction module. The channel separation operation can be expressed as: Among them, f channel-split (·) represents the channel separation operation, and x i represents the output of the i-th channel separation pixel attention residual module; represents the feature sent to the next channel separation pixel attention residual module for processing after channel separation, represents the feature sent to the bypass feature extraction module for processing; in particular: Among them, represents the input of the first channel-separated pixel attention residual module, without bypass feature processing, directly connected to the output of the last channel-separated pixel attention residual module in the channel dimension; The bypass feature processing module achieves feature extraction simply and efficiently by connecting the bypass features in the channel dimension. The operations in the bypass feature processing module can be expressed as: Among them, I Bypass represents the total output of the bypass feature extraction module, and f c (,) represents concatenating feature maps along the channel dimension; The operations within the 16-channel separated pixel attention residual module can be expressed as: Among them, I Trunk represents the output of the 16th channel-separated pixel attention residual module; After the input of the channel-separated pixel attention residual module is channel-separated, part of the features are processed by the pixel attention module, and the other part of the features pass through a convolution layer with only one convolution kernel size of 3×3, and a residual connection is used inside the channel-separated pixel attention residual module; The total output of the deep feature extraction module can be expressed as:

2. The infrared image super-resolution reconstruction method based on channel-separated deep neural network according to claim 1, characterized in that The specific method of preparing the training data set includes: For the collected public infrared image dataset, the bicubic interpolation downsampling algorithm is used to simulate the image degradation process, and low-resolution infrared images corresponding to scale factors s=2, s=3 and s=4 are obtained respectively. The above process can be expressed as: Among them, I HR represents the existing high-resolution infrared image, and f bicubic (·) represents the bicubic interpolation downsampling operation, which is the corresponding low-resolution infrared image obtained by downsampling with a scale factor of s.

3. The infrared image super-resolution reconstruction method based on channel separation deep neural network according to claim 1, characterized in that: The image reconstruction module includes a feature reconstruction module and a convolution layer with a convolution kernel size of 1×1. The operations in the image reconstruction module can be expressed as: I IR = f 1×1 (f FR (I DFE )) Among them, f FR (·) represents the internal operation of the feature reconstruction module; At this point, the final output of the network can be expressed as: I SR = f nearest (I LR ) + I IR Among them, f nearest (·) represents nearest neighbor interpolation, and I SR represents the finally reconstructed high-resolution infrared image.

4. The infrared image super-resolution reconstruction method based on channel-separated deep neural network according to claim 1, characterized in that The specific methods for training the constructed deep neural network model include: Determine the optimizer, learning rate, and maximum number of iterations used for training, and start training the network; after each forward propagation is completed, calculate the L1 loss between I SR and I LR , and then continue to update the network parameters through the backpropagation algorithm; repeat the above steps until the preset maximum number of iterations is reached, thereby completing the network training.

5. The infrared image super-resolution reconstruction method based on a channel-separated deep neural network according to claim 1, characterized in that The specific method of saving the model includes: solidifying a set of network weights corresponding to the highest evaluation index in the evaluation process of network training, saving them as the final network model, and using them for later testing to reconstruct high-resolution infrared images.

Images