Infrared image restoration model training method adaptive to embedded platform

Abstract

The infrared image restoration model training method suitable for the embedded platform of the application comprises the following steps: 1) constructing an infrared image restoration large model, training the same by using the data of actually collected real low-resolution infrared images and real high-resolution infrared images; 2) constructing a portable infrared image restoration small model suitable for the embedded platform; and 3) setting the adjustment coefficient to 0 at the beginning of training of the infrared image restoration small model, and then increasing the value along with the training process, and setting the adjustment coefficient to 1 at the end of training. The application gradually changes the loss function along with the training process, directly and quickly approaches the optimal solution of the infrared image restoration convex optimization problem at the beginning of training, then gradually increases the disturbance of high-frequency details in the loss function to make the solution of the convex optimization problem perform gradient descent near the optimal solution, and gradually increases the real high-frequency information in the solution without deviating from the optimal solution.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to a method for training an infrared image restoration model adapted to an embedded platform. Background Technology

[0002] Due to its unique imaging method, infrared technology can clearly image objects even at night by accepting their thermal radiation, making it widely used in various fields such as outdoor observation. Because of these applications, infrared thermal imagers require real-time imaging and are highly integrated into embedded devices with complex optical systems. In the long and complex imaging process of infrared thermal imagers, they inevitably suffer from image degradation problems such as noise and blurring. Furthermore, due to limitations in infrared detector technology, most mainstream infrared thermal imagers currently have a resolution of around 640×512, also suffering from low-resolution degradation. Traditional methods typically involve a series of image processing algorithms, including temporal denoising, spatial denoising, dynamic range adjustment, enhancement, local contrast improvement, and interpolation magnification, to try to mitigate these degradations and achieve good image quality, but the results are ultimately unsatisfactory.

[0003] With the rise of deep learning methods in recent years, infrared thermal imager manufacturers have gradually replaced some parts of the above process with deep learning models, achieving relatively good results. However, because the embedded platforms used in infrared thermal imagers cannot support high-performance NPUs, the deep learning models used must be relatively small in terms of parameters, memory usage, and running speed. Only smaller deep learning models can be successfully ported. This leads to a trade-off between the performance and size of infrared image restoration models. Knowledge distillation is used to improve the performance of smaller models within the scope of portable model size. Knowledge distillation methods can generally be divided into target distillation and feature distillation. Target distillation typically uses the inference results of a larger model to establish a loss function, allowing the smaller model to learn from the inference results of the larger model. Feature distillation usually requires the large and small models to have the same structure, allowing supervised learning between the feature vectors output by some modules. Both methods improve the performance of the smaller model to some extent, but they also have some problems:

[0004] Knowledge distillation can lead to smaller models overly relying on the inference results of larger models, neglecting to learn the truth and resulting in overly ambiguous outcomes. Furthermore, if the truth is learned directly, the limited capabilities of small infrared image reconstruction models make it difficult to learn stable optimal solutions, and the reconstruction results may instead fluctuate excessively based on the rich high-frequency textures of the truth. Summary of the Invention

[0005] Therefore, the technical problem to be solved by the present invention is to overcome the defects in the prior art and to propose an infrared image restoration model training method adapted to embedded platforms.

[0006] To address the aforementioned technical problems, this invention provides a method for training an infrared image restoration model adapted to an embedded platform, comprising:

[0007] A method for training an infrared image restoration model adapted to an embedded platform includes the following steps:

[0008] 1) Construct a large-scale infrared image reconstruction model using real, low-resolution infrared images acquired in real-world scenarios. and true high-resolution infrared images The data was used to train it.

[0009] 2) Construct a portable, embedded-platform-compatible small-scale infrared image restoration model, with the loss function being:

[0010] ,

[0011] in, A dynamically changing loss function satisfying the convex function property is constructed for a small model of infrared image restoration. A small model for infrared image reconstruction. For L1 norm, the optimal solution Input a real low-resolution infrared image into the large-scale infrared image reconstruction model. High-resolution infrared image restored by super-resolution reconstruction. To match the actual low-resolution infrared image The corresponding true high-resolution infrared image, For the optimal solution The low-resolution infrared image obtained after downsampling. This is an adjustment coefficient that dynamically changes the loss function. ;

[0012] 3) Adjust the coefficients at the beginning of training the small infrared image reconstruction model. Set it to 0, and then increase the adjustment factor as the training process progresses. The value will be adjusted by the coefficient at the end of the training period. Set to 1.

[0013] As one of the feasible solutions For the optimal solution The low-resolution infrared image obtained after bilinear interpolation downsampling.

[0014] As one possible implementation, the average of the restoration results from multiple attempts using the trained large-scale infrared image restoration model is taken as the optimal solution. .

[0015] As one feasible implementation, the infrared image restoration large model is a classic super-resolution reconstruction method in the visible light field.

[0016] As one possible implementation scheme, the infrared image reconstruction mini-model includes,

[0017] The dimensionality reduction module performs spatial dimensionality reduction on the input low-resolution infrared image to obtain a new low-resolution infrared image. And extract its shallow feature vector ;

[0018] Multiple consecutive feature restoration modules are used. Each module first feeds the input feature vector into multiple consecutive convolutional layers and activation functions to obtain a progressively restored high-dimensional feature vector. Then, the progressively restored high-dimensional feature vector is fed into the convolutional layers, and residual and attention weight mechanisms are executed sequentially to obtain the output high-dimensional feature vector of the feature restoration module. The input feature vector is a shallow feature vector. Or the high-dimensional feature vector output by the previous feature restoration module;

[0019] The image reconstruction module, and the final feature restoration module, output high-dimensional feature vectors and low-resolution infrared images. Feature stitching is performed, and the stitched features are convolved and upsampled to obtain the final restoration result of the infrared image restoration mini-model.

[0020] As one possible implementation, the adjustment coefficient is increased linearly during the training process. The value of .

[0021] The technical solution of this invention has the following advantages:

[0022] The infrared image restoration model training method adapted to embedded platforms of this invention constructs a loss function that gradually changes during the training process. In the early stages of training, it directly and quickly approximates the optimal solution of the convex optimization problem in infrared image restoration. Then, during training, it gradually adds high-frequency detail perturbations to the loss function, causing gradient descent of the solution near the optimal solution. Without deviating from the optimal solution, it gradually adds real high-frequency information to the solution. This allows for rapid approximation of the optimal solution while obtaining high-fidelity infrared image restoration results with real high-frequency information. Thus, the small infrared image restoration model achieves better restoration results than knowledge distillation training methods, improving the image quality of the corresponding deep learning model presented on embedded devices. The processing method of this invention uses a large infrared image restoration model to approximate the optimal solution while smoothing detail textures. Through the training mode proposed in this method, it can learn the effect of the large infrared image restoration model as much as possible while adding more real high-frequency details, achieving results superior to general knowledge distillation methods. Attached Figure Description

[0023] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the logical structure of the present invention.

[0025] Figure 2 This is a schematic diagram of the network structure of a small model for infrared image reconstruction.

[0026] Figure 3 This is a schematic diagram of the feature restoration module structure.

[0027] Figure 4 The image shows a comparison of the restoration results. Column a represents the original low-resolution infrared image before processing, column b represents the 2x super-resolution result of the large-scale infrared image restoration model, and column c represents the 2x super-resolution result of the small-scale infrared image restoration model after processing according to this invention. Detailed Implementation

[0028] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] This method can be applied to infrared image restoration problems with convex loss functions. Image restoration problems generally include denoising, deblurring, and super-resolution reconstruction. The network structures of these restoration problems are not significantly different. Therefore, this invention will illustrate the restoration problem using super-resolution reconstruction. The corresponding process of this method is as follows: Figure 1 As shown.

[0030] The present invention provides a method for training an infrared image restoration model adapted to an embedded platform, comprising the following steps:

[0031] 1) Construct a large-scale infrared image reconstruction model using real, low-resolution infrared images acquired in real-world scenarios. and true high-resolution infrared images For training a large-scale infrared image restoration model, this invention does not limit the network structure, nor does it consider the model's inference time or the number of parameters. It only uses the model's image processing quality performance as the sole criterion, allowing for multiple choices in constructing such a model. In the visible light domain, there are many classic super-resolution reconstruction methods, such as SwinIR, HAT, and Real-ESRGAN, all of which have demonstrated good results in the visible light domain and can be applied to infrared image processing. This invention uses the HAT network structure as the large-scale infrared image restoration model, only modifying the input channels of its shallow feature extraction module and the output channels of its super-resolution reconstruction module to 1 to adapt to infrared images. Therefore, this invention will not provide a detailed description of the HAT network structure, but mainly introduces the process of constructing and training the infrared dataset. For SwinIR, when processing infrared images, in addition to the necessary modification of the number of input channels to 1 to adapt to the single-channel characteristics of infrared images, since infrared images generally have low resolution and a lot of noise, the ECSSR model designed based on SwinIR can also introduce a cross-shaped window self-attention module to increase the receptive field, use local enhanced position coding to obtain position information, and add an edge detection auxiliary network to enhance edge information, etc. to improve the super-resolution reconstruction effect of infrared images. These will not be elaborated here.

[0032] Specifically, this invention uses two thermal imagers, one high-resolution and one low-resolution, to acquire images of a real scene. Then, it performs image registration on the acquired real high-resolution and low-resolution infrared images to construct a real low-resolution infrared image. True high-resolution infrared images The constructed dataset is used to train a large-scale infrared image restoration model. Correspondingly, the high-resolution infrared thermal imager has a resolution of 1280×1024, and the low-resolution infrared thermal imager has a resolution of 640×512. Using the above method, this invention constructs a realistic infrared super-resolution dataset for training the large-scale infrared image restoration model by capturing data from different scenes. Next, this invention constructs a loss function using the above dataset to train the large-scale infrared image restoration model. The specific loss function is shown below:

[0033] (1)

[0034] in, These are real low-resolution infrared images and real high-resolution infrared images from the real infrared dataset constructed in this invention, with image sizes of 640×512 and 1280×1024, respectively. This invention is a large-scale infrared image restoration model after modifying the shallow feature extraction module and the super-resolution reconstruction module in the classic super-resolution reconstruction model HAT. To train a large-scale model for infrared image reconstruction The loss function used, The loss function is L1 norm. Using the above loss function, this invention trains the model on a constructed real infrared dataset, and outputs a large-scale infrared image restoration model after fitting the loss function. This prepares for the training of subsequent small models, providing effective guidance for optimal solutions in the early stages of training, reducing interference from high-frequency detail textures on the training process, and facilitating rapid approximation of the infrared image restoration convex optimization problem.

[0035] 2) Constructing a portable, embedded-platform-compatible small-scale infrared image restoration model. This invention focuses on constructing a small-scale infrared image restoration model with portability as its key consideration, enabling high-frame-rate imaging on thermal imagers. Therefore, this invention does not employ complex network structures for the small-scale infrared image restoration model; instead, it uses a convolutional neural network with simple activation functions to design the model, maximizing performance while maintaining portability. The specific network structure is as follows: Figure 2 As shown in the diagram. First, for the input low-resolution infrared image, it is first input into the spatial dimension reduction module to reduce its spatial dimension and thus reduce the computational load. This embodiment takes 2x super-resolution reconstruction as an example. To achieve 2x super-resolution reconstruction, the spatial dimension reduction module performs downsampling by 2x, and then the final image reconstruction module performs upsampling by 4x, thereby achieving the overall goal of 2x super-resolution reconstruction.

[0036] As a specific embodiment, the infrared image restoration mini-model includes a dimensionality reduction module to obtain multiple feature information required for restoration super-resolution reconstruction, such as three consecutive feature restoration modules and an image reconstruction module.

[0037] The dimension reduction module is used to process the input low-resolution infrared image, such as a real low-resolution infrared image. Low-resolution infrared images are obtained by reducing spatial dimensions. And extract its shallow feature vector , specifically,

[0038] (2)

[0039] in, The input images are a real low-resolution infrared image and a low-resolution infrared image with the spatial dimension broken down into the channel dimension, respectively, with sizes of 1×640×512 and 4×320×256. This refers to the `torch.nn.PixelUnShuffle` function in PyTorch. The scale for downsampling in the spatial dimension is taken as... The low-resolution infrared images input to the infrared image restoration mini-model are taken from the aforementioned real infrared dataset during the training phase. After being loaded onto the embedded platform, the low-resolution infrared images input to it are the real degraded infrared images acquired by the infrared device.

[0040] Low-resolution infrared images are obtained by breaking down the spatial dimensional information of real low-resolution infrared images into channel dimensions. Then, it is input into the shallow feature extraction module to extract shallow feature vectors, as shown below:

[0041] (3)

[0042] in, The extracted shallow feature vector has a size of 48×320×256; It is a 3×3 convolution operation with 4 input channels and 48 output channels.

[0043] Three consecutive feature restoration modules with shallow feature vectors The initial input is processed sequentially to recover the feature information needed for super-resolution reconstruction, as shown below:

[0044] (4)

[0045] (5)

[0046] (6)

[0047] in, The high-dimensional feature vectors output sequentially by the three consecutive feature restoration modules are all 48×320×256 in size. These are three consecutive feature restoration modules, all with the same network structure, as shown in the diagram below. Figure 3 As shown. Therefore, each feature restoration module will not be described in detail here, only the following will be discussed. Please provide a detailed explanation.

[0048] First, the extracted shallow feature vectors Two consecutive convolutional layers and activation functions are input to perform high-resolution information restoration for the feature restoration module. and They are respectively the feature restoration module and Output high-dimensional feature vector , The general processing steps for the input feature vector are as follows:

[0049] (7)

[0050] (8)

[0051] in, These are two high-dimensional feature vectors, one and two, representing the stepwise reconstruction of high-resolution spatial information. Both are 48×320×256 in size. `relu` is the `torch.nn.relu` activation function in PyTorch. Both are 3×3 convolution operations, with 48 input and 48 output channels.

[0052] Next, the high-dimensional feature vector is divided into two... After inputting into the convolutional layer, the residual mechanism and attention weight mechanism are executed sequentially to obtain the high-dimensional feature vector output by the feature restoration module. As shown below:

[0053] (9)

[0054] in, The input feature vectors for this module are shallow feature vectors and high-dimensional feature vectors, respectively, both of which are 48×320×256 in size; This refers to the torch.nn.relu activation function in PyTorch. This is a 3×3 convolution operation with 48 input and 48 output channels.

[0055] The third feature restoration module is obtained after the third consecutive feature restoration module. The output is a high-dimensional feature vector with restored high-resolution spatial information. .

[0056] The image reconstruction module will use high-dimensional feature vectors Low-resolution infrared images with spatial dimensions scattered across channel dimensions After feature concatenation, convolution, and upsampling, the final restoration result of this infrared image restoration mini-model is output, as shown below:

[0057] (10)

[0058] in, The final restoration result output by the infrared image restoration small model of the present invention has a size of 1×1280×1024. The operation involves concatenating features along the feature dimension. This refers to the `torch.nn.PixelShuffle` function in PyTorch. The scale parameter for upsampling is taken in this embodiment. . It is a 3×3 convolution operation with 52 input channels and 4 output channels.

[0059] To quickly approximate the optimal solution to this convex optimization problem in the early stages of training, the loss function is used to limit the perturbation of real high-frequency information on model training. The specific loss function is as follows:

[0060] (11)

[0061] in, A dynamically changing loss function satisfying the convex function property is constructed for a small model of infrared image restoration. This invention presents a small model for infrared image restoration. It is an L1 norm. These are real low-resolution infrared images from the previously constructed real infrared dataset. True high-resolution infrared images The image sizes are 640×512 and 1280×1024 respectively; the optimal solution The real low-resolution infrared image is used as input for the large-scale infrared image reconstruction model mentioned above. High-resolution infrared image restored by super-resolution reconstruction. For the optimal solution The low-resolution infrared image obtained after bilinear interpolation downsampling, in this invention specifically refers to 2x downsampling; To mix low-resolution infrared images Input to infrared image restoration mini-model The output is a restored high-resolution infrared image; This is an adjustment coefficient that dynamically changes the loss function. , The closer a value is to 0, the closer the loss function is to the optimal solution. It has a strong approximation capability. The closer the value is to 1, the stronger the loss function's ability to approximate the true high-resolution infrared image (HR).

[0062] In super-resolution reconstruction, due to the abundance of high-frequency detail information in real high-resolution infrared images, high-resolution reconstruction from low-resolution infrared images struggles to fully and faithfully reconstruct these details. Furthermore, after each super-resolution reconstruction, the reconstructed high-resolution infrared image exhibits slight variations and randomness in its high-frequency detail information. This invention uses the average of multiple reconstruction results from the aforementioned large-scale infrared image reconstruction model as the optimal solution to the convex optimization problem of infrared image reconstruction. For example, the average of five results is used to remove the influence of high-frequency detail perturbations. This invention utilizes the output of the trained large-scale infrared image reconstruction model as the optimal solution to the convex optimization problem of the small-scale infrared image reconstruction model. This allows the model to quickly and directly approach the optimal solution in the early stages of training. During training, perturbations of real high-frequency details are gradually added to the loss function, causing gradient descent of the solution to the convex optimization problem near the optimal solution. Without deviating from the optimal solution, real high-frequency information is gradually added to the solution, thereby increasing the stability of the small-scale infrared image reconstruction model training and effectively increasing the high-frequency information of the real high-resolution infrared image, resulting in a high-fidelity reconstruction.

[0063] Therefore, for the loss function that satisfies the convex function property used in the small infrared image restoration model constructed above, this invention uses the trained and fitted large infrared image restoration model, with the real low-resolution infrared image as its input, and the output result as the optimal solution, and substitutes it into the above loss function to obtain the following loss function, as shown below:

[0064] (12)

[0065] in, The above steps use a real super-resolution infrared rate dataset to train a fitted infrared image restoration model. To reconstruct a large model from pre-trained infrared images using real low-resolution infrared images To input a high-resolution infrared image that has been reconstructed and restored using super-resolution technology, This is to obtain a low-resolution infrared image by performing 2x bilinear interpolation downsampling on the high-resolution infrared image output by the pre-trained infrared image reconstruction model. It is an L1 norm; This invention presents a small model for infrared image restoration. For mixed low-resolution infrared images Input to infrared image restoration mini-model The output is a restored high-resolution infrared image.

[0066] That is, in the training of the infrared image restoration mini-model, this invention uses a mixed image as input for training, and the input image is... Its corresponding truth value is This constitutes the training data pairs for the small model of infrared image reconstruction.

[0067] 3) Adjust the coefficients at the beginning of training the small infrared image reconstruction model. Set it to 0, and then increase it as the training process progresses, such as linearly increasing it. The value will be adjusted by the coefficient at the end of the training period. Set to 1. By constructing the above training data pairs, this invention dynamically adjusts the coefficients during the training process. This allows for dynamic adjustment of the direction and intensity of gradient descent during training, aiming to quickly approximate the optimal solution in the early stages of training. Then, during subsequent training, perturbations of real high-frequency details are gradually added to the loss function to ensure gradient descent approaches the optimal solution of the convex optimization problem, achieving a high-fidelity restoration. The coefficients are adjusted during training. The logic for dynamic adjustment is as follows: at the start of training, this invention will adjust the coefficients. Set it to 0, and then gradually increase it linearly as the training process progresses. The value is then adjusted at the end of training. Set to 1.

[0068] This specific embodiment of the invention provides a specific setting method: In this specific embodiment, the epoch for training the small model is set to 200. When the epoch < 20, the following settings are provided: ;exist When, set ;exist hour, The value increases linearly from 0 to 1.

[0069] The processing result is as follows Figure 4As shown, column a represents the original low-resolution infrared image before processing, with a resolution of 640×512. It suffers from the inherent detail blurring of low-resolution images, as seen in the boxed areas, where edges, scene textures are unclear, and pixelation is noticeable. It may also be accompanied by slight noise common in infrared imaging. Column b represents the 2x super-resolution result of the large-scale infrared image restoration model, with a resolution of 1280×1024. Column c represents the 2x super-resolution result of the small-scale infrared image restoration model after processing, with a resolution of 1280×1024. This is the final processing result obtained by using a dynamic loss function and the optimal solution of the large-scale model. A comparative analysis is as follows:

[0070] 1) After processing with the small model, the blocky texture of the image is significantly eliminated, the overall size is enlarged and the pixel arrangement is regular, and the resolution is consistent with the large model processing result b.

[0071] 2) It retains more details than large models and incorporates real high-frequency information. Because the training of large models focuses on fitting the overall optimal solution, it will filter out some high-frequency details of real infrared images, such as the edge contours of small targets in the scene, resulting in a smoother and blurrier image.

[0072] The small model uses a dynamic loss function to accurately preserve the high-frequency details of real infrared images while maintaining resolution.

[0073] 3) High fidelity, effectively avoids over-reproduction, closely resembles real scenes, and the processed image has no false details or blurring.

[0074] The essence of this invention's infrared image processing effect is to achieve a balance between resolution improvement, detail preservation, and fidelity under the premise of adapting a small model to embedded devices. It not only solves the problem of unclear original low-resolution images, but also overcomes the defects of insufficient detail in large models and instability of traditional small models. The final output infrared image can meet the clarity requirements of the human eye or subsequent analysis, and also conforms to the realism of infrared imaging. At the same time, because the small model design has low computing power requirements, it can adapt to the real-time requirements of devices.

[0075] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A method for training an infrared image restoration model adapted to an embedded platform, characterized in that, The method comprises the following steps of: 1) Constructing an infrared image restoration large model, using real collected real low-resolution infrared images and real high-resolution infrared images to train it, 2) constructing a portable infrared image restoration small model adapted to the embedded platform, and a loss function thereof is ， Wherein, The loss function dynamically changed for the infrared image restoration small model satisfies the convex function property, The infrared image restoration small model, The L1 norm, the optimal solution The infrared image restoration large model inputs the real low-resolution infrared image The high-resolution infrared image restored by super-resolution reconstruction, The real low-resolution infrared image, The corresponding real high-resolution infrared image, The optimal solution The low-resolution infrared image obtained after downsampling, The adjustment coefficient of the dynamically changed loss function, ; 3) Adjust the coefficients at the beginning of training the small infrared image reconstruction model. Set it to 0, and then increase the adjustment factor as the training process progresses. The value will be adjusted by the coefficient at the end of the training period. Set to 1.

2. The method of claim 1, wherein the infrared image restoration model training method is adapted to an embedded platform. For the optimal solution of bilinear interpolation down-sampling the low-resolution infrared image obtained after 3. The method of claim 2, wherein the infrared image restoration model is trained for an embedded platform. The mean of multiple restoration results of the trained infrared image restoration large model is taken as the optimal solution .

4. The method of claim 3, wherein the infrared image restoration model training method is adapted to an embedded platform. The infrared image restoration large model is a classical super-resolution reconstruction method in the visible light field.

5. The method of claim 1, wherein the infrared image restoration model training method is adapted to an embedded platform. The infrared image restoration small model comprises a dimensionality reduction module that performs spatial dimensionality reduction on the input low resolution infrared image to obtain a low resolution infrared image and extract its shallow feature vector ; A plurality of continuous feature restoration modules, each feature restoration module first inputs an input feature vector into a plurality of continuous convolutional layers and an activation function to obtain a step-by-step restored high-dimensional feature vector; then inputs the step-by-step restored high-dimensional feature vector into a convolutional layer and sequentially executes a residual mechanism and an attention weight mechanism to obtain a high-dimensional feature vector output by the feature restoration module, wherein the input feature vector is a shallow layer feature vector or a high-dimensional feature vector output by a previous feature restoration module An image reconstruction module, the high-dimensional feature vector output by the final feature restoration module and the low-resolution infrared image Feature splicing is performed, the spliced features are convolved and up-sampled to obtain the final restoration result of the infrared image restoration small model.

6. The method of claim 1, wherein, The value of the adjustment coefficient is linearly increased with the training process .

Images