Thermal image super-resolution method based on noise squeezing and channel attention mechanism

Abstract

The application discloses a thermal imaging image super-resolution method based on noise extrusion and channel attention mechanism, and comprises the following steps: acquiring a to-be-reconstructed thermal imaging image; and performing image reconstruction on the to-be-reconstructed thermal imaging image by using a super-resolution network to obtain a super-resolution thermal imaging image; wherein the super-resolution network is a trained super-resolution network obtained by training through thermal imaging training data; and the super-resolution network comprises a feature extraction module, a multi-level channel attention module, a noise extrusion module, a high-frequency filtering module and an up-sampling module. The application improves the detail imaging effect of the super-resolution thermal image reconstruction based on the proposed new super-resolution network model.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a super-resolution method for thermal imaging images based on noise compression and channel attention mechanisms. Background Technology

[0002] Thermal imaging technology refers to the use of infrared detectors and optical imaging lenses to receive the infrared radiation energy distribution pattern of a target object and reflect it onto the photosensitive element of the infrared detector, thereby obtaining an infrared thermal image. This thermal image corresponds to the heat distribution field on the object's surface. Common infrared thermal imagers convert the invisible infrared energy emitted by an object into a visible thermal image, with different colors on the thermal image representing different temperatures of the object. Single-image super-resolution is a typical ill-posed problem in computer vision and remains an active research area.

[0003] The goal of single-image super-resolution is to recover an accurate high-resolution representation from a given low-resolution image. Over the past two decades, various methods have been proposed, but in recent years, new deep learning-based techniques have made significant progress. Most deep learning neural network-based methods are primarily used in the visible spectrum domain. For thermal images, resolution is a common limitation, related to the sensor technology used. The use of these images in various applications across multiple fields (such as medicine, security, and industry) is increasing. Therefore, super-resolution techniques for thermal imaging images have become an urgent need in the industry, which places new demands on the imaging quality and denoising capabilities of super-resolution. Most existing super-resolution reconstruction methods for thermal imaging combine segmentation maps to achieve the thermal imaging super-resolution reconstruction task. These methods use pixel-level feature information from visible light image segmentation to compensate for the detail information lost during thermal imaging super-resolution reconstruction, effectively enhancing the contour information of the super-resolution image and improving its detail.

[0004] However, such methods require the use of features learned from visible light images in segmentation tasks to optimize the features learned from thermal imaging images during super-resolution, which greatly complicates data acquisition. At the same time, since visible light and thermal imaging images are not acquired by the same sensor, there is a deviation (imaging error) between the two images from the beginning, requiring additional registration. Otherwise, the loss of learned feature differences will have a negative impact on the reconstruction effect. Summary of the Invention

[0005] To address the aforementioned problems in the existing technology, this invention provides a super-resolution method for thermal imaging based on noise compression and channel attention mechanisms. The technical problem to be solved by this invention is achieved through the following technical solution:

[0006] This invention provides a super-resolution method for thermal imaging images based on noise compression and channel attention mechanisms, comprising:

[0007] Acquire the thermal imaging image to be reconstructed;

[0008] A super-resolution thermal imaging image is obtained by reconstructing the thermal imaging image to be reconstructed using a super-resolution network.

[0009] The super-resolution network is a trained super-resolution network obtained by training on thermal imaging training data; the super-resolution network includes a feature extraction module, a multi-level channel attention module, a noise compression module, a high-frequency filtering module, and an upsampling module, wherein...

[0010] The feature extraction module is used to extract features from the thermal imaging image to be reconstructed; the multi-channel attention module is used to enhance the extracted features by multiple channels and extract detailed features based on all enhancements; the noise squeezing module is used to calculate feature similarity and denoise the detailed features; the high-frequency filtering module is used to extract high-frequency information from the thermal imaging image to be reconstructed; and the upsampling module is used to stitch the denoised information and the high-frequency information together and perform upsampling reconstruction processing on the stitched result to obtain the super-resolution thermal imaging image.

[0011] In one embodiment of the present invention, the feature extraction module includes a plurality of convolutional blocks and a first convolutional layer connected in sequence, wherein the input of the first convolutional block is also connected to an adder with the output of the last convolutional block; each convolutional block includes a plurality of second convolutional layers and a first convolutional layer connected in sequence, wherein the input of the first second convolutional layer is connected to an adder with the output of the last second convolutional layer.

[0012] In one embodiment of the present invention, the multi-level channel attention module includes N parallel enhancement channels, an adder connected to the outputs of the N enhancement channels, and a multiplier connected to the output of the adder and the inputs of the N enhancement channels, where N is an integer greater than 0.

[0013] The nth enhancement channel includes n parallel third convolutional layers and one convolutional activation layer, where n takes values ​​from 1 to N;

[0014] For enhancement channels n>1, each enhancement channel also includes a dot multiplier whose input is connected to the outputs of n third convolutional layers, and whose output is connected to the input of the convolutional activation layer.

[0015] In one embodiment of the present invention, the process of feature similarity calculation and denoising in the noise compression module includes:

[0016] Determine the front channel used for feature similarity calculation;

[0017] Calculate the feature similarity loss corresponding to the front channel; wherein, the feature similarity loss is used for updating the network parameters of the super-resolution network during training;

[0018] Noise reduction is performed on the post-channels that are not used for feature similarity calculation;

[0019] The output of the noise compression module is obtained by splicing the output of the pre-channel with the output of the noise-reduced post-channel.

[0020] In one embodiment of the present invention, the high-frequency filtering module includes a plurality of fourth convolutional layers and high-pass filters connected in sequence, and an adder connected to the input of the first fourth convolutional layer and the output of the last high-pass filter, the output of which is connected to a fifth convolutional layer.

[0021] In one embodiment of the present invention, the high-pass filter employs a low-frequency filter inversion strategy.

[0022] In one embodiment of the present invention, the upsampling module employs a subpixel convolution strategy; wherein the input of the upsampling module is connected to a fully connected layer.

[0023] In one embodiment of the present invention, the training process of the super-resolution network includes:

[0024] Acquire thermal imaging training data; the thermal imaging training data includes thermal imaging label images, as well as noise-free low-resolution thermal imaging images and noisy low-resolution thermal imaging images corresponding to the thermal imaging label images;

[0025] For each iteration, the process includes: inputting the noise-free low-resolution thermal imaging image and the noisy low-resolution thermal imaging image into the feature extraction module to extract features, obtaining a first extracted feature and a second extracted feature; inputting the first extracted feature and the second extracted feature into the multi-level channel attention module to extract detail features, obtaining a first detail feature and a second detail feature; simultaneously inputting the first detail feature and the second detail feature into the noise compression module for denoising and feature similarity loss calculation; inputting the noisy low-resolution thermal imaging image into the high-frequency filtering module to extract high-frequency information; concatenating the denoised information and the high-frequency information and inputting them into the upsampling module for upsampling reconstruction; calculating the image output loss based on the upsampling reconstruction result and the thermal imaging label image; updating the network parameters of the super-resolution network based on the feature similarity loss and the image output loss; and repeating the above process under the updated super-resolution network until the maximum number of iterations is reached to obtain a trained super-resolution network.

[0026] In one embodiment of the present invention, acquiring thermal imaging training data includes:

[0027] Acquire thermal imaging tag images;

[0028] The thermal imaging tag image is downsampled to obtain a noise-free low-resolution thermal imaging image corresponding to the thermal imaging tag image;

[0029] Noise is added to the noise-free low-resolution thermal imaging image to form a corresponding noisy low-resolution thermal imaging image.

[0030] In one embodiment of the present invention, the training process of the super-resolution network further includes:

[0031] After each iteration of upsampling reconstruction, the PSNR of the corresponding upsampling reconstruction result is calculated;

[0032] Find the largest PSNR among all the sampled reconstruction results, and use the super-resolution network corresponding to the one with the largest output PSNR as the trained super-resolution network.

[0033] The beneficial effects of this invention are:

[0034] This invention proposes a thermal imaging super-resolution method based on noise squeezing and channel attention mechanisms. Instead of using visible light to compensate for lost details during thermal imaging super-resolution reconstruction, it proposes a novel super-resolution network model. This model incorporates a noise squeezing module, a high-pass filtering module, and a multi-level channel attention mechanism. The noise squeezing module effectively removes noise from the thermal imaging image to be reconstructed. A high-frequency filtering module extracts high-frequency information from the original input, and the multi-level attention mechanism extracts high-frequency information from the features. Through residual connections, the high-frequency information extracted by the high-frequency filtering module is merged with the high-frequency information learned by the network, and then used for super-resolution upsampling to improve the output index score and the reconstruction detail. This invention does not introduce visible light during the training and reconstruction of the super-resolution network model; it only uses thermal imaging data. This effectively avoids the need for additional registration processes and, while ensuring convenient access to thermal imaging data, avoids reconstruction losses caused by thermal imaging errors due to visible light, thus enhancing image details and contour information.

[0035] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0036] Figure 1This is a flowchart illustrating a thermal imaging super-resolution method based on noise compression and channel attention mechanism provided in an embodiment of the present invention.

[0037] Figure 2 This is a schematic diagram of the structure of a super-resolution network for thermal image reconstruction provided in an embodiment of the present invention;

[0038] Figure 3 This is a schematic diagram of the feature extraction module provided in an embodiment of the present invention;

[0039] Figure 4 This is a schematic diagram of the structure of the multi-level channel attention module provided in an embodiment of the present invention;

[0040] Figure 5 This is a schematic diagram of the data processing process of the noise suppression module provided in an embodiment of the present invention;

[0041] Figure 6 This is a schematic diagram of the structure of the high-frequency filtering module provided in an embodiment of the present invention;

[0042] Figure 7 This is a schematic diagram of the high-pass filter in the high-frequency filtering module provided in this embodiment of the invention;

[0043] Figure 8 This is a schematic diagram of the network structure corresponding to the training process of the super-resolution network provided in the embodiments of the present invention;

[0044] Figure 9 This is a schematic diagram of the training process of the super-resolution network provided in an embodiment of the present invention;

[0045] Figure 10 This is a schematic diagram of another training process of the super-resolution network provided in this embodiment of the invention;

[0046] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0047] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0048] Example 1

[0049] To enhance the contour information of super-resolution images and improve their detail, i.e., the reconstruction effect of detail information. Please see... Figure 1 This invention provides a super-resolution method for thermal imaging images based on noise compression and channel attention mechanisms, specifically including the following steps:

[0050] S10. Obtain the thermal imaging image to be reconstructed; the thermal imaging image to be reconstructed is a low-resolution thermal imaging image, denoted as X.

[0051] S20. Using a super-resolution network, image reconstruction is performed on the thermal imaging image to be reconstructed to obtain a super-resolution thermal imaging image; wherein, the super-resolution network is a trained super-resolution network obtained by training on thermal imaging training data; please refer to [link to relevant documentation]. Figure 2 The super-resolution network provided in this embodiment of the invention includes a feature extraction module, a multi-channel attention module, a noise compression module, a high-frequency filtering module, and an upsampling module.

[0052] In this embodiment of the invention, thermal imaging training data is first acquired before training. To better achieve the training of the high-resolution network, corresponding noise-free low-resolution thermal imaging images and noisy low-resolution thermal imaging images are generated based on existing thermal imaging label images. The thermal imaging label images are super-resolution images. Specifically, acquiring thermal imaging training data includes:

[0053] The thermal imaging tag image HR is acquired; HR is a super-resolution image. The thermal imaging tag image HR is then downsampled to obtain a noise-free low-resolution thermal imaging image corresponding to HR. Here, the thermal imaging tag image HR is downsampled by a factor of 4 to obtain a low-resolution image, which is then used as the noise-free low-resolution thermal imaging image. For noise-free low-resolution thermal imaging images Noise is added or blurred to generate the corresponding noisy low-resolution thermal imaging image (LR). Throughout the training process, the thermal imaging label image (HR) and its corresponding noise-free low-resolution thermal imaging image are used. The super-resolution network was trained using noisy, low-resolution thermal imaging images (LR).

[0054] Next, we will provide a detailed introduction to the implementation process of each part of the super-resolution network.

[0055] The feature extraction module is used to extract features from the thermal imaging image to be reconstructed; the feature extraction result is denoted as F. in .

[0056] This invention provides an optional scheme where the feature extraction module includes a plurality of convolutional blocks and a first convolutional layer connected in sequence. The input of the first convolutional block and the output of the last convolutional block are connected to an adder. Each convolutional block includes a plurality of second convolutional layers and a first convolutional layer connected in sequence. The outputs of the first and last second convolutional layers are connected to an adder. It can be seen that there are skip connections of a residual network between convolutional blocks, and there are also skip connections of a residual network within each convolutional block.

[0057] For example, please see Figure 3 The feature extraction module of this embodiment includes three convolutional blocks connected in sequence and a 1×1 first convolutional layer. The input of the first convolutional block and the output of the third convolutional block are connected to an adder for superposition. Each convolutional block includes two 3×3 second convolutional layers and a 1×1 first convolutional layer connected in sequence. The input of the first second convolutional layer and the output of the second convolutional layer are connected to an adder for superposition.

[0058] Furthermore, the multi-channel attention module in this embodiment of the invention is used to enhance the extracted features through multiple channels, and extract detailed features based on all enhanced features; the extracted detailed features are denoted as F. out .

[0059] Specifically, in this embodiment of the invention, the multi-channel attention module includes N parallel enhancement channels, an adder connected to the outputs of the N enhancement channels, and a multiplier connected to the output of the adder and the inputs of the N enhancement channels, where N is an integer greater than 0.

[0060] The nth enhancement channel includes n parallel third convolutional layers and one convolutional activation layer, where n takes values ​​from 1 to N;

[0061] For enhancement channels n>1, each enhancement channel also includes a dot multiplier whose input is connected to the outputs of n third convolutional layers, and whose output is connected to the input of the convolutional activation layer.

[0062] For example, please see Figure 4 The multi-channel attention module of this invention includes three parallel enhancement channels, which respectively enhance the features F extracted by the feature extraction module. in Perform one, two, and three enhancements, and extract detailed features F based on the results of the three enhancements. out Its extraction of detail features F out Represented as:

[0063] F out =S R ⊙F in (1)

[0064] Among them, F out F represents the final output of the multi-channel attention module. in This indicates the features extracted by the feature extraction module. exist Figure 4 Chinese As shown, Sigmoid is a non-linear activation function. F i (X i ) in F iThis represents a non-linear activation function, such as the non-linear ReLU activation function. Typically, a 1×1 convolutional layer follows the ReLU activation function. Figure 4 The outputs of F1, F2, and F3, where ⊙ represents the element-wise product, and X... i This represents the output corresponding to the dot product of the outputs of the i-th enhancement channel and the third convolutional layer. This represents the output of the j-th third convolutional layer in the i-th enhancement channel.

[0065] As can be seen, the embodiments of the present invention utilize a multi-level channel attention module to further learn the high-frequency information of the input image and extract features containing more high-frequency information.

[0066] Furthermore, the noise squeezing module in this embodiment of the invention is used to calculate the feature similarity of detailed features and to denoise them; the denoising result is denoted as S. out .

[0067] Specifically, in this embodiment of the invention, the detailed features output by the multi-level channel attention module are input into the noise squeezing module for feature similarity calculation and denoising. The specific process includes:

[0068] The first channel used for feature similarity calculation is determined; the feature similarity loss corresponding to the first channel is calculated; the feature similarity loss is used to update the network parameters of the super-resolution network during training; the second channel not used for feature similarity calculation is denoised; the output corresponding to the first channel is concatenated with the output corresponding to the denoised second channel to obtain the output of the noise squeezing module.

[0069] Please see Figure 5 , Figure 5 This schematically illustrates how to optimize the noise squeezing module based on detailed features F. out Perform feature similarity calculation and noise reduction.

[0070] For the training phase, Figure 5 This includes the noisy low-resolution thermal imaging image (LR) output by the upper branch using a multi-channel attention module, and the noise-free low-resolution thermal imaging image output by the lower branch using a multi-channel attention module. The impact of noise on image reconstruction is evaluated using corresponding detailed features. The features of the first n1 channels are fed into a similarity loss function for loss calculation and backpropagation. The last n2 channels are used as reserved channels to squeeze noise into them. Here, n = n1 + n2, where n is a power of 2. To minimize the value of the noise channel n2, thereby reducing the impact of noise on the feature channel n1, in this embodiment, n1 is relatively large and n2 is relatively small. For example, if the detailed features output by the multi-level channel attention module include 64 channels (n = 64), then n1 can be 60, and the detailed features corresponding to the remaining n2 = 4 channels are considered as noise.

[0071] The similarity loss function during the training phase is expressed as follows:

[0072] L cos =1-cos(E1,E2) (2)

[0073] Among them, L cos Let E1 represent the features of the noisy low-resolution thermal image (LR) in the first n1 channels of the noise compression module, and E2 represent the features of the noisy low-resolution thermal image (LR) in the first n1 channels of the noise compression module. The similarity loss is calculated using E1 and E2 to approximate the similarity between E1 and E2. Simultaneously, the noise contained in the original features can be squeezed into the last n2 channels. and here He was dropped because he did not participate in further training.

[0074] Next, the features E1 of the first n1 channels of the noisy low-resolution thermal imaging image LR in the noise compression module are stitched together with the denoising results of the last n2 channels (not used for feature similarity calculation). The formula is expressed as:

[0075]

[0076] Among them, S out This represents the noise reduction result of the noise compression module, where γ represents the noise figure when 0 < γ < 1. This represents the characteristics of a noisy low-resolution thermal image (LR) in the n2 channels following the noise compression module, where cat indicates the connection operation. The goal is to reduce the impact of noise on the low-resolution thermal imaging (LR) image, by extracting features from the input features of the noise compression module for the last n² channels. The noise reduction process is performed to obtain the noise reduction result.

[0077] For the reconstruction phase, only Figure 5The upper branch, as shown, utilizes the detailed features F corresponding to the thermal imaging image X to be reconstructed, output by a multi-channel attention module. out The values ​​of the first n1 channels and the last n2 channels in the reconstruction process are the same as those in the training stage. That is, if n1 = 60 and n2 = 4 in the training stage, then n1 = 60 and n2 = 4 in the reconstruction stage. The same method as in the training stage is used for denoising, except that the reconstruction stage does not need to use the lower branch and does not need to calculate the feature similarity loss.

[0078] As can be seen, this embodiment of the invention utilizes a noise compression module to squeeze noise in low-resolution images into a very small number of channels of the feature map (considered as inefficient channels) through the learning of the convolutional neural network. Then, feature similarity is calculated for the effective channels, and the inefficient channels are blurred. These processes enable the super-resolution network to have better capabilities in extracting effective features and denoising.

[0079] Furthermore, in this embodiment of the invention, the high-frequency filtering module is used to extract high-frequency information from the thermal imaging image to be reconstructed; the extracted high-frequency information is denoted as Y. out .

[0080] Specifically, embodiments of the present invention employ a convolution-filter-convolution-filter structure, and then superimpose the obtained high-frequency information with the input via a skip connection. An alternative embodiment of the present invention provides a high-frequency filtering module comprising several sequentially connected fourth convolutional layers and high-pass filters, and an adder connected to the input of the first fourth convolutional layer and the output of the last high-pass filter; the output of the adder is connected to a fifth convolutional layer.

[0081] for example Figure 6 In the high-frequency filtering module, there are two fourth convolutional layers and a high-pass filter connected in sequence, namely, a 3×3 fourth convolutional layer, a high-pass filter, a 3×3 fourth convolutional layer, and a high-pass filter connected in sequence. The input of the first 3×3 fourth convolutional layer and the output of the second high-pass filter are connected to an adder, and the output of the adder is connected to a 1×1 fifth convolutional layer.

[0082] The high-pass filter employs a strategy of inverting the low-frequency filter. Please see [link / reference]. Figure 7 This invention provides an optional approach, where the filtering weights are determined using G(X). Here, G(X) includes a 3×3 convolutional layer, a normalization layer (BN), and a softmax layer. First, a series of feature maps of the input data X are extracted using a 3×3 convolutional layer. Then, the feature maps are divided into G groups using a softmax layer, with each group having K channels. 2 That is, each pixel position (i,j) corresponds to a K×K vector, i.e. Figure 7 In the small block region of ω, the low-pass filter weight ω determined in this embodiment of the inventioni,j The formula is expressed as:

[0083] ω i,j =G(X) i,j (4)

[0084] Each group can perform spatial adaptive anti-aliasing, and all groups combined can achieve channel grouping adaptive anti-aliasing in this embodiment of the invention.

[0085] For spatial adaptive anti-aliasing, we first obtain feature maps of G×K×K channels. Then, each pixel position (i,j) corresponds to a K×K vector, which is transformed into a K×K kernel and applied to the thermal imaging image X to be reconstructed. The filtered output at pixel position (i,j) can be expressed by the formula:

[0086]

[0087] Among them, Y i,j This represents the low-pass filter output at pixel position (i,j), where Ω refers to a region surrounding pixel position (i,j). The values ​​p and q determine the size of the Ω region. This represents the filter weight at pixel position (i,j), where the filter size is p×q, and X... i+p,j+q This represents the pixel value of the p×q region at pixel position (i,j).

[0088] Applying the above operations to the adaptive anti-aliasing of channel groups in this embodiment of the invention, the filtering output result of each group g, where 1≤g≤G, can be expressed by the following formula:

[0089]

[0090] Among them, Y i,j,g This represents the low-pass filter output at pixel position (i,j) in the g-th group. This represents the filter weight at pixel position (i,j) in the g-th group, where the filter weight size is p×q, and X i+p,j+q,g This represents the pixel value of the p×q region at pixel position (i,j) in the g-th group.

[0091] The low-pass filtered outputs of the G groups are concatenated to obtain the following: Figure 7 The low-pass filter output Y is shown. After obtaining this low-pass filter output Y, subtracting Y from X reverses the low-pass filter result (Reverse in the diagram indicates reversal) to obtain the high-frequency information, i.e., the final output Y' of the high-pass filter. out =XY.

[0092] To balance performance and computational cost, the same weights are used for different channels of features belonging to the same group within the features grouped into G. This is because the feature channels are also similar, so they can share the filter weight coefficients.

[0093] Y' out It was just a symbolic representation. Figure 6 The calculation method for the output of a certain high-pass filter, the whole Figure 6 The implementation process completes the high-frequency information Y of the high-frequency filtering module. out Extraction.

[0094] Here, the input X of the high-frequency filtering module is the feature obtained after the thermal imaging image to be reconstructed is passed through a 3×3 convolutional layer.

[0095] As can be seen, the embodiments of the present invention employ a learnable high-frequency filtering module to first extract and retain high-frequency information from the input original low-resolution image, which is then used to enhance the high-frequency features in subsequent upsampling reconstruction, thereby improving the reconstruction of detailed features.

[0096] Furthermore, in this embodiment of the invention, the upsampling module is used to stitch together the denoised information and high-frequency information, and to perform upsampling reconstruction processing on the stitched result to obtain a super-resolution thermal imaging image.

[0097] Specifically, the output Y of the high-frequency filtering module out and the output S of the noise compression module out The splicing is performed, and the splicing result is shown as follows:

[0098] UP in =cat(Y out ,S out (7)

[0099] And UP in Transformed into UP' through a linear fully connected layer. in The data is then input into the upsampling module, where sub-pixel convolution is used. The upsampling result is represented as follows:

[0100] SR = UP(UP' in (8)

[0101] SR refers to the super-resolution thermal imaging image obtained through reconstruction processing.

[0102] As can be seen, the embodiments of the present invention combine the high-frequency information extracted by the high-frequency filtering module with the high-frequency information learned by the multi-level channel attention module in the main body of the network through residual connection, and jointly perform super-resolution upsampling operation to improve the index score and imaging effect of the super-resolution image output.

[0103] based on Figure 2 The super-resolution network shown is used to construct a twin super-resolution network for training, as follows: Figure 8 As shown, Figure 9 The corresponding training process includes the following steps:

[0104] S801. Acquire thermal imaging training data; the thermal imaging training data includes thermal imaging label images, as well as noise-free low-resolution thermal imaging images and noisy low-resolution thermal imaging images corresponding to the thermal imaging label images.

[0105] During training, noise-free low-resolution thermal imaging images are used. The noisy low-resolution thermal imaging image LR is input into the twin super-resolution network. The twin network is characterized by having the same network structure and shared network parameters.

[0106] Training involves multiple iterations, with the maximum number of iterations set according to design requirements, for example, 1000 here. Each iteration includes:

[0107] S802, respectively, noise-free low-resolution thermal imaging images The noisy, low-resolution thermal imaging image (LR) is input into the feature extraction module to extract features, resulting in the first extracted feature and the second extracted feature.

[0108] For details, please see [link to relevant documentation]. Figure 9 The noise-free low-resolution thermal imaging images were respectively... And noisy low-resolution thermal imaging images LR input such as Figure 3 The feature extraction module shown performs feature extraction to obtain the first extracted feature and the second extracted feature. Among them, The module that performs feature extraction with LR uses the same network parameters, that is, it shares network parameters.

[0109] S803. Input the first extracted feature and the second extracted feature into the multi-level channel attention module respectively to extract the first detail feature and the second detail feature.

[0110] For details, please see [link to relevant documentation]. Figure 9 The first extracted feature and the second extracted feature are respectively input as follows: Figure 4 The multi-channel attention module shown extracts first and second detail features. The modules that extract detail features from the first and second extracted features use the same network parameters, i.e., they share network parameters.

[0111] S804. Simultaneously, the first detail feature and the second detail feature are input into the noise compression module for denoising processing, and feature similarity loss is calculated.

[0112] For details, please see [link to relevant documentation]. Figure 9 Simultaneously, the first and second detail features are input into, for example... Figure 5 The noise compression module shown performs denoising and calculates feature similarity loss. After training, the determined front channel n1 and back channel n2 are used not only for denoising and feature similarity loss calculation in each iteration, but also for the reconstruction of subsequent thermal imaging images.

[0113] S805: Input the noisy low-resolution thermal imaging image (LR) to the high-frequency filtering module to extract high-frequency information.

[0114] For details, please see [link to relevant documentation]. Figure 9 Inputting noisy, low-resolution thermal imaging images into a system such as LR Figure 6 and Figure 7 The high-frequency filtering module shown extracts high-frequency information. Here, before the noisy low-resolution thermal imaging image (LR) is input into the high-frequency filtering module, the features obtained after passing through a 3×3 convolutional layer are then input into the high-frequency filtering module.

[0115] S806: The denoised information and high-frequency information are spliced ​​together and input into the upsampling module for upsampling reconstruction processing.

[0116] Specifically, following the above processing, the denoised information and high-frequency information are spliced ​​together. After the spliced ​​information passes through a fully connected layer, it is input to the upsampling module to obtain the upsampling reconstruction result corresponding to the current iteration.

[0117] S807. Calculate the image output loss based on the upsampling reconstruction result and the thermal imaging tag image. The image output loss calculated in the current iteration can be expressed as: L = L1(HR,SR1), where HR represents the thermal imaging tag image and SR1 represents the upsampling reconstruction result of the current iteration corresponding to the noisy low-resolution thermal imaging image LR.

[0118] S808. Update the network parameters of the super-resolution network based on feature similarity loss and image output loss. When updating the super-resolution network parameters, the feature similarity loss of the noise squeezing module and the image output loss of the upsampling module are taken into account, making the network parameter update of the super-resolution network more accurate.

[0119] After the super-resolution network is updated, the above S801 to S808 processes are repeated under the updated super-resolution network until the maximum number of iterations is reached, then the following steps are executed:

[0120] S809. Use the super-resolution network corresponding to the network parameters at this time as the trained super-resolution network.

[0121] Here, in order to accelerate training, the thermal imaging training data is preprocessed: the thermal imaging training data is cut into blocks according to a specified block size, and the pixel size within the block is normalized before being used as the input thermal imaging training data for the super-resolution network.

[0122] Furthermore, the research revealed that using this maximum number of iterations method may result in a suboptimal super-resolution network. Similarly, designing the stopping condition by setting a preset threshold for network loss may also lead to a suboptimal super-resolution network. To address these issues, this invention provides an alternative solution, please refer to [link to alternative solution]. Figure 10 The training process of the super-resolution network in this embodiment of the invention further includes:

[0123] After each iteration of upsampling and reconstruction processing,

[0124] S810. Calculate the PSNR corresponding to the upsampling reconstruction result; specifically, calculate the PSNR value between the upsampling reconstruction result SR1 and the thermal imaging tag image HR in the current iteration. The PSNR value can be calculated using existing methods and will not be elaborated here. Each iteration corresponds to one PSNR value.

[0125] S811. Find the largest PSNR among all the PSNRs corresponding to the upsampling reconstruction results, and use the super-resolution network corresponding to the output with the largest PSNR as the trained super-resolution network. Through research and experimental verification by the inventors, the largest PSNR usually corresponds to the optimal reconstruction effect. Therefore, this embodiment of the invention finds the largest PSNR among all the PSNRs corresponding to the upsampling reconstruction results, identifies which iteration corresponds to the largest PSNR, and uses the super-resolution network corresponding to that iteration as the trained super-resolution network for subsequent reconstruction of thermal imaging images.

[0126] In summary, the thermal imaging super-resolution method based on noise squeezing and channel attention mechanisms proposed in this invention does not use visible light to compensate for lost details during thermal imaging super-resolution reconstruction. Instead, it proposes a novel super-resolution network model that incorporates a noise squeezing module, a high-pass filtering module, and a multi-level channel attention mechanism. The noise squeezing module effectively removes noise from the thermal imaging image to be reconstructed. A high-frequency filtering module extracts high-frequency information from the original input, and a multi-level attention mechanism extracts high-frequency information from the features. Residual connections are used to merge the high-frequency information extracted by the high-frequency filtering module with the high-frequency information learned by the network, performing a combined super-resolution upsampling operation to improve the output index score and reconstruction detail imaging effect of the super-resolution image. This invention does not introduce visible light during the training and reconstruction of the super-resolution network model; it only uses thermal imaging data, effectively avoiding the need for additional registration processes. While ensuring convenient access to thermal imaging data, it also avoids reconstruction losses caused by thermal imaging errors due to the introduction of visible light, thereby enhancing image details and contour information.

[0127] Please see Figure 11 This invention provides an electronic device, including a processor 1101, a communication interface 1102, a memory 1103, and a communication bus 1104, wherein the processor 1101, the communication interface 1102, and the memory 1103 communicate with each other through the communication bus 1104.

[0128] Memory 1103 is used to store computer programs;

[0129] When the processor 1101 executes the program stored in the memory 1103, it implements the steps of the above-described thermal imaging super-resolution method based on noise compression and channel attention mechanism.

[0130] This invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described thermal imaging super-resolution method based on noise compression and channel attention mechanisms.

[0131] For the embodiments of the device / electronic device / storage medium, since they are basically similar to the method embodiments, the description is relatively simple, and relevant parts can be referred to in the description of the method embodiments.

[0132] In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0133] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0134] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A super-resolution method for thermal imaging images based on noise compression and channel attention mechanisms, characterized in that, include: Acquire the thermal imaging image to be reconstructed; A super-resolution thermal imaging image is obtained by reconstructing the thermal imaging image to be reconstructed using a super-resolution network. The super-resolution network is a trained super-resolution network obtained by training on thermal imaging training data; the super-resolution network includes a feature extraction module, a multi-level channel attention module, a noise compression module, a high-frequency filtering module, and an upsampling module, wherein... The feature extraction module is used to extract features from the thermal imaging image to be reconstructed; the multi-channel attention module is used to enhance the extracted features by multiple channels and extract detailed features based on all enhancements; the noise squeezing module is used to calculate feature similarity and denoise the detailed features; the high-frequency filtering module is used to extract high-frequency information from the thermal imaging image to be reconstructed; and the upsampling module is used to stitch the denoised information and the high-frequency information together and perform upsampling reconstruction processing on the stitched result to obtain the super-resolution thermal imaging image.

2. The thermal imaging super-resolution method based on noise compression and channel attention mechanism according to claim 1, characterized in that, The feature extraction module includes a plurality of convolutional blocks and a first convolutional layer connected in sequence. The input of the first convolutional block is also connected to an adder with the output of the last convolutional block. Each convolutional block includes a plurality of second convolutional layers and a first convolutional layer connected in sequence. The input of the first second convolutional layer is connected to an adder with the output of the last second convolutional layer.

3. The thermal imaging super-resolution method based on noise compression and channel attention mechanism according to claim 1, characterized in that, The multi-level channel attention module includes N parallel enhancement channels, an adder connected to the outputs of the N enhancement channels, and a multiplier connected to the output of the adder and the inputs of the N enhancement channels, where N is an integer greater than 0. The nth enhancement channel includes n parallel third convolutional layers and one convolutional activation layer, where n takes values ​​from 1 to N; For enhancement channels n>1, each enhancement channel also includes a dot multiplier whose input is connected to the outputs of n third convolutional layers, and whose output is connected to the input of the convolutional activation layer.

4. The thermal imaging super-resolution method based on noise compression and channel attention mechanism according to claim 1, characterized in that, The noise compression module performs feature similarity calculation and denoising processes, including: Determine the front channel used for feature similarity calculation; Calculate the feature similarity loss corresponding to the front channel; wherein, the feature similarity loss is used for updating the network parameters of the super-resolution network during training; Noise reduction is performed on the post-channels that are not used for feature similarity calculation; The output of the noise compression module is obtained by splicing the output of the pre-channel with the output of the noise-reduced post-channel.

5. The thermal imaging super-resolution method based on noise compression and channel attention mechanism according to claim 1, characterized in that, The high-frequency filtering module includes several fourth convolutional layers and high-pass filters connected in sequence, as well as an adder connected to the input of the first fourth convolutional layer and the output of the last high-pass filter. The output of the adder is connected to the fifth convolutional layer.

6. The thermal imaging super-resolution method based on noise compression and channel attention mechanism according to claim 5, characterized in that, The high-pass filter employs a low-frequency filter inversion strategy.

7. The thermal imaging super-resolution method based on noise compression and channel attention mechanism according to claim 1, characterized in that, The upsampling module employs a subpixel convolution strategy; wherein, the input of the upsampling module is connected to a fully connected layer.

8. The thermal imaging super-resolution method based on noise compression and channel attention mechanism according to claim 1, characterized in that, The training process of the super-resolution network includes: Acquire thermal imaging training data; the thermal imaging training data includes thermal imaging label images, as well as noise-free low-resolution thermal imaging images and noisy low-resolution thermal imaging images corresponding to the thermal imaging label images; For each iteration, the process includes: inputting the noise-free low-resolution thermal imaging image and the noisy low-resolution thermal imaging image into the feature extraction module to extract first and second extracted features; inputting the first and second extracted features into the multi-level channel attention module to extract first and second detail features; simultaneously inputting the first and second detail features into the noise compression module for denoising and feature similarity loss calculation; inputting the noisy low-resolution thermal imaging image into the high-frequency filtering module to extract high-frequency information; concatenating the denoised information and the high-frequency information into the upsampling module for upsampling reconstruction; calculating the image output loss based on the upsampling reconstruction result and the thermal imaging label image; updating the network parameters of the super-resolution network based on the feature similarity loss and the image output loss; repeating the above process under the updated super-resolution network until the maximum number of iterations is reached to obtain a trained super-resolution network.

9. The thermal imaging super-resolution method based on noise compression and channel attention mechanism according to claim 8, characterized in that, The acquisition of thermal imaging training data includes: Acquire thermal imaging tag images; The thermal imaging tag image is downsampled to obtain a noise-free low-resolution thermal imaging image corresponding to the thermal imaging tag image; Noise is added to the noise-free low-resolution thermal imaging image to form a corresponding noisy low-resolution thermal imaging image.

10. The thermal imaging super-resolution method based on noise compression and channel attention mechanism according to claim 8, characterized in that, The training process of the super-resolution network also includes: After each iteration of upsampling reconstruction, the PSNR of the corresponding upsampling reconstruction result is calculated; Find the largest PSNR among all the sampled reconstruction results, and use the super-resolution network corresponding to the one with the largest output PSNR as the trained super-resolution network.

Images