A deep learning-based unpaired infrared image enhancement method
By employing a deep learning approach with multi-scale encoders and attention mechanisms, the problem of unpaired data in infrared image enhancement is solved, achieving brightness restoration, texture enhancement, and noise suppression, and generating high-quality images suitable for infrared target detection and tracking.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ARMY ENG UNIV OF PLA
- Filing Date
- 2026-01-19
- Publication Date
- 2026-05-29
AI Technical Summary
Existing infrared image enhancement methods rely on paired high- and low-quality training samples, which are difficult to apply effectively without paired data. This results in low brightness, poor contrast, and loss of texture details in infrared images. Furthermore, traditional methods are prone to introducing noise or artifacts during the enhancement process.
A multi-scale encoder is used to extract infrared image features, which are combined with dense residual blocks, channel residual blocks and deformable convolutional residual blocks, and channel and spatial attention mechanisms. The network is trained by self-supervised reconstruction loss, perceptual loss and infrared noise contrast loss to achieve unpaired infrared image enhancement.
It significantly improves image brightness and texture details, suppresses background noise, and generates high-contrast, low-noise enhanced images without the need for high-resolution paired images, making it suitable for infrared target detection and tracking tasks.
Smart Images
Figure CN122115259A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a deep learning-based unpaired infrared image enhancement method, specifically a method for infrared image enhancement using deep learning under unpaired data conditions, belonging to the fields of infrared image processing and deep learning technology. Background Technology
[0002] Infrared imaging technology, with its passive detection and all-weather capabilities, is widely used in night vision surveillance, military reconnaissance, and medical imaging. However, limited by sensor noise, weak thermal radiation, and environmental interference, infrared images often exhibit low brightness, poor contrast, and loss of texture detail. Traditional infrared image enhancement methods, such as histogram equalization, Retinex, or filtering techniques, often introduce noise or artifacts while improving brightness.
[0003] Existing deep learning-based image enhancement methods have achieved good results in the visible light domain, but most rely on paired low / high-quality training samples. However, high-quality paired data is difficult to obtain in real-world infrared scenes, limiting their engineering applications. Therefore, to address these issues, a paired-free infrared image enhancement method is urgently needed. Summary of the Invention
[0004] To address the aforementioned issues, this invention proposes a deep learning-based unpaired infrared image enhancement method that enables network training based solely on low-quality infrared images to output high-contrast, low-noise enhanced images.
[0005] The unpaired infrared image enhancement method based on deep learning of the present invention extracts multi-level features of infrared images through a multi-scale encoder, enhances high-frequency textures using dense residual blocks, highlights high-radiation areas using channel residual blocks, and adapts to target deformation using deformable convolutional residual blocks; at the same time, it combines channel and spatial attention mechanisms to suppress background clutter; during the training process, self-supervised reconstruction loss, perceptual loss, and infrared noise contrast loss are introduced to ensure that the network can still restore brightness, enhance details, and suppress noise even without a reference image; it includes the following steps: Step 1: Data preprocessing, Acquiring low-quality infrared images I in Linear normalization and pseudo-color mapping are applied to the image to scale pixel values to 0 and 1 and expand it into a three-channel input. Step 2: Network construction Construct a multi-scale self-supervised augmentation network consisting of encoder E, residual module R, attention module A, and decoder D: a. Encoder E contains four downsampling convolutional layers Conv1 to Conv4 in sequence. Each convolutional layer has a kernel size of 3×3 and a stride of 2. The number of output channels is 64, 128, 256 and 512 in sequence. Conv4 is followed by a four-scale pyramid pooling module to extract the global context. b. The residual module R is stacked sequentially with dense residual blocks, channel residual blocks and deformable convolutional residual blocks. The dense residual blocks are composed of three layers of convolutional units and use cross-layer connections to enhance high-frequency textures. The channel residual blocks embed Squeeze-Excitation channel attention in the residual path to highlight high-radiation areas. The deformable convolutional residual blocks use learnable offsets Δp to adjust the convolution sampling position to adapt to target deformation. c. Attention module A includes a parallel structure of channel attention (CA) and spatial attention (SA), and their calculation formulas are as follows: , ; In the formula, GAP is global average pooling, δ is the ReLU function, and σ is the Sigmoid function; d. Decoder D contains four upsampling layers, each followed by a 3×3 convolution and skip connections to progressively recover high-resolution features and fuse them with the encoder output; Step 3: Network training, When using only low-quality infrared images, a network is jointly optimized using self-supervised reconstruction loss, perception loss, and infrared noise contrast loss. The overall loss function is: ; in, ; In the formula, L rec The self-supervised reconstruction loss is used to constrain the network output image I. out With input low-quality infrared image I in Consistency in structural information; no high-quality reference images are required during training; L per To perceive loss, L con For noise contrast loss; construct positive sample patches p containing different noise characteristics in the same infrared image. i With negative sample patch n i The network is guided to bring structurally consistent regions closer together and distance noise-dominant regions from each other in the feature space, thereby suppressing non-Gaussian noise caused by detector inconsistencies and thermal noise during infrared imaging; SSIM is the structural similarity index, I outTo output the enhanced result, Φ(•) represents the output features of the pre-trained VGG-19 network conv3_3 layer, s(•,•) represents the cosine similarity, τ represents the temperature parameter, and f i This is the feature representation of the i-th local region of the image extracted by encoder E; Step 4: Image enhancement Input the infrared image to be enhanced into the trained network, and output the enhancement result I. out The output pixel values are limited to the range of 0 and 1 by using the tanh function.
[0006] Furthermore, each convolutional layer in the encoder E includes batch normalization and LeakyReLU activation to stabilize training and prevent gradient vanishing.
[0007] Furthermore, the residual module R contains 3 to 5 dense residual blocks, 3 to 5 channel residual blocks, and 3 to 5 deformable convolutional residual blocks, with a total of no less than 9.
[0008] Furthermore, the perceived loss L per The weights λ² and the self-supervised reconstruction loss L rec Weight λ1 and noise contrast loss L con The weights λ3 are set in a ratio of λ1:λ2:λ3 = 2:1:0.5 to balance brightness restoration and detail enhancement. The self-supervised reconstruction loss L... rec As the main loss term, it is used to ensure the stability of the overall structure and brightness distribution of the images before and after enhancement; perceptual loss L per Used to supplement high-level semantic and detailed information; noise contrast loss L con As an auxiliary constraint, it can effectively suppress non-Gaussian noise in infrared imaging with a small weight, thereby achieving a balance between detail enhancement and noise suppression while ensuring structural consistency.
[0009] Furthermore, the convolution output of the deformable convolutional residual block satisfies: , In the formula, ω k p is the convolution weight, p0 is the reference coordinate corresponding to the current convolution position in the output feature map, p k The coordinates of the standard convolution sampling points, Δp k This is a learnable offset.
[0010] Compared with existing technologies, the deep learning-based unpaired infrared image enhancement method of the present invention does not require high-definition paired images, which greatly reduces the cost of training data acquisition; the synergistic effect of multiple types of residual blocks and dual attention mechanism effectively enhances texture details and suppresses background noise; the average gradient and information entropy of the output image are significantly improved, and it can be directly used for infrared target detection and tracking tasks. Attached Figure Description
[0011] Figure 1 This is a schematic diagram of the network structure framework of the present invention.
[0012] Figure 2 This is a schematic diagram of the training process of the present invention.
[0013] Figure 3 This is a schematic diagram comparing infrared images before and after enhancement according to the present invention.
[0014] Figure (a) shows the original infrared image, and Figure (b) shows the enhanced image generated by the method of the present invention. Detailed Implementation
[0015] like Figure 1 and Figure 2 The deep learning-based unpaired infrared image enhancement method shown extracts multi-level features from infrared images through a multi-scale encoder, enhances high-frequency textures using dense residual blocks, highlights high-radiation areas using channel residual blocks, and adapts to target deformation using deformable convolutional residual blocks. Simultaneously, it combines channel and spatial attention mechanisms to suppress background clutter. During training, self-supervised reconstruction loss, perceptual loss, and infrared noise contrast loss are introduced to ensure that the network can still restore brightness, enhance details, and suppress noise even without a reference image. The method includes the following steps: Step 1: Data preprocessing, Acquiring low-quality infrared images I in Linear normalization and pseudo-color mapping are applied to the image to scale pixel values to 0 and 1 and expand it into a three-channel input. Step 2: Network construction Construct a multi-scale self-supervised augmentation network consisting of encoder E, residual module R, attention module A, and decoder D: a. The encoder E consists of four downsampling convolutional layers Conv1 to Conv4, each with a kernel size of 3×3 and a stride of 2. The number of output channels is 64, 128, 256, and 512 respectively. Conv4 is followed by a four-scale pyramid pooling module to extract the global context. Each convolutional layer in the encoder E is followed by batch normalization and LeakyReLU activation to stabilize training and prevent gradient vanishing. b. The residual module R sequentially stacks dense residual blocks, channel residual blocks, and deformable convolutional residual blocks. The dense residual blocks consist of three convolutional units connected across layers to enhance high-frequency textures. The channel residual blocks embed Squeeze-Excitation channel attention within the residual path to highlight high-radiation regions. The deformable convolutional residual blocks utilize a learnable offset Δp to adjust the convolutional sampling position to adapt to target deformation. The convolutional output of the deformable convolutional residual blocks satisfies the following: , In the formula, ω k p is the convolution weight, p0 is the reference coordinate corresponding to the current convolution position in the output feature map, p k The coordinates of the standard convolution sampling points, Δp k For learnable offsets; The residual module R contains 3 to 5 dense residual blocks, 3 to 5 channel residual blocks, and 3 to 5 deformable convolutional residual blocks, with a total of no less than 9. c. Attention module A includes a parallel structure of channel attention (CA) and spatial attention (SA), and their calculation formulas are as follows: , ; In the formula, GAP is global average pooling, δ is the ReLU function, and σ is the Sigmoid function; d. Decoder D contains four upsampling layers, each followed by a 3×3 convolution and skip connections to progressively recover high-resolution features and fuse them with the encoder output; Step 3: Network training, When using only low-quality infrared images, a network is jointly optimized using self-supervised reconstruction loss, perception loss, and infrared noise contrast loss. The overall loss function is: ; in, ; In the formula, L rec The self-supervised reconstruction loss is used to constrain the network output image I. out With input low-quality infrared image I in Consistency in structural information; no high-quality reference images are required during training; L per To perceive loss, L con For noise contrast loss; construct positive sample patches p containing different noise characteristics in the same infrared image. i With negative sample patch n iThe network is guided to bring structurally consistent regions closer together and distance noise-dominant regions from each other in the feature space, thereby suppressing non-Gaussian noise caused by detector inconsistencies and thermal noise during infrared imaging; SSIM is the structural similarity index, I out To output the enhanced result, Φ(•) represents the output features of the pre-trained VGG-19 network conv3_3 layer, s(•,•) represents the cosine similarity, τ represents the temperature parameter, and f i This is the feature representation of the i-th local region of the image extracted by encoder E; Wherein, the perceived loss L per The weights λ² and the self-supervised reconstruction loss L rec Weight λ1 and noise contrast loss L con The weight λ3 is set to a ratio of λ1:λ2:λ3=2:1:0.5 to balance brightness restoration and detail enhancement. The self-supervised reconstruction loss L... rec As the main loss term, it is used to ensure the stability of the overall structure and brightness distribution of the images before and after enhancement; perceptual loss L per Used to supplement high-level semantic and detailed information; noise contrast loss L con As an auxiliary constraint, it can effectively suppress non-Gaussian noise in infrared imaging with a small weight, thereby achieving a balance between detail enhancement and noise suppression while ensuring structural consistency. Step 4: Image enhancement Input the infrared image to be enhanced into the trained network, and output the enhancement result I. out The output pixel values are limited to the range of 0 and 1 by using the tanh function.
[0016] Example 1: The deep learning-based unpaired infrared image enhancement method of the present invention includes the following steps: First, the input infrared image I... in Linear normalization is performed and the input is expanded to a three-channel input. Next, multi-scale features are extracted by encoder E, which contains four 3×3 convolutional downsampling layers and a pyramid pooling module at the end to obtain the global context. Then, feature enhancement is performed by residual module R, which includes dense residual blocks, channel residual blocks, and deformable convolutional residual blocks, stacked sequentially to form a deep feature representation. Key region responses are then enhanced by channel attention and spatial attention modules. Next, decoder D restores the image spatial resolution and generates the enhanced result through four levels of deconvolutional upsampling and skip connections. Finally, a joint loss function L is used for training, where self-supervised reconstruction loss ensures structural consistency, perceptual loss maintains global visual quality, and noise contrast loss improves local contrast and suppresses infrared-specific non-Gaussian noise.
[0017] The specific operating method is as follows: First, taking a 256×256 resolution infrared image as an example, the pixel values are normalized to 0 and 1 and expanded into a three-channel input.
[0018] Secondly, encoder E employs four downsampling convolutions, each with a kernel size of 3×3, a stride of 2, and channel numbers of 64, 128, 256, and 512, respectively. The residual module R consists of three dense residual blocks, three channel residual blocks, and three deformable convolutional residual blocks stacked sequentially. Each dense residual block contains three convolutional units connected across layers. The channel residual blocks introduce a Squeeze-Excitation structure in the residual path. The deformable convolutional residual blocks achieve adaptive sampling by learning the offset Δp. The attention module A contains a parallel structure of channel attention and spatial attention. The decoder D contains four levels of deconvolutional upsampling, each followed by a 3×3 convolution and skipped to the corresponding encoder output.
[0019] During the training phase, the Adam optimizer was used with an initial learning rate of 1×10^-4, a batch size of 16, and a training duration of 50 to 100 epochs. The loss weights were set to λ1:λ2:λ3=2:1:0.5. Finally, as Figure 3 As shown, the trained network can output enhanced results with increased brightness, clearer details, and reduced noise on unseen infrared images; its average gradient is increased by about 35%, information entropy is increased by about 20%, and the average detection rate in infrared target detection tasks is increased by about 10%.
[0020] This invention presents a deep learning-based unpaired infrared image enhancement method. It achieves brightness restoration, texture enhancement, and noise suppression of infrared images even in the absence of high-resolution reference images. This is achieved by constructing a multi-scale encoder-decoder network and introducing dense residual blocks, channel residual blocks, and deformable convolutional residual blocks into the network, combined with a dual channel and spatial attention mechanism. During training, a self-supervised reconstruction loss, perceptual loss, and infrared noise contrast loss are jointly optimized, allowing the network to converge even using only low-quality infrared images. This method generates high-contrast, low-noise enhanced infrared images without requiring paired high-resolution data and can be directly applied to subsequent tasks such as infrared target detection, recognition, and tracking.
[0021] The above embodiments are merely preferred embodiments of the present invention. Therefore, all equivalent changes or modifications made to the structure, features and principles described in the claims of this patent application are included within the scope of this patent application.
Claims
1. A deep learning-based unpaired infrared image enhancement method, characterized in that, Includes the following steps: Step 1: Data preprocessing, Acquiring low-quality infrared images I in Linear normalization and pseudo-color mapping are applied to the image to scale pixel values to 0 and 1 and expand it into a three-channel input. Step 2: Network construction Construct a multi-scale self-supervised augmentation network consisting of encoder E, residual module R, attention module A, and decoder D: a. Encoder E contains four downsampling convolutional layers Conv1 to Conv4 in sequence. Each convolutional layer has a kernel size of 3×3 and a stride of 2. The number of output channels is 64, 128, 256 and 512 in sequence. Conv4 is followed by a four-scale pyramid pooling module to extract the global context. b. The residual module R is stacked sequentially with dense residual blocks, channel residual blocks and deformable convolutional residual blocks. The dense residual blocks are composed of three layers of convolutional units and use cross-layer connections to enhance high-frequency textures. The channel residual blocks embed Squeeze-Excitation channel attention in the residual path to highlight high-radiation areas. The deformable convolutional residual blocks use learnable offsets Δp to adjust the convolution sampling position to adapt to target deformation. c. Attention module A includes a parallel structure of channel attention (CA) and spatial attention (SA), and their calculation formulas are as follows: ， ； In the formula, GAP is global average pooling, δ is the ReLU function, and σ is the Sigmoid function; d. Decoder D contains four upsampling layers, each followed by a 3×3 convolution and skip connections to progressively recover high-resolution features and fuse them with the encoder output; Step 3: Network training, When using only low-quality infrared images, a network is jointly optimized using self-supervised reconstruction loss, perception loss, and infrared noise contrast loss. The overall loss function is: ； in, ； In the formula, L rec The self-supervised reconstruction loss is used to constrain the network output image I. out With input low-quality infrared image I in Consistency in structural information; no high-quality reference images are required during training; L per To perceive loss, L con For noise contrast loss; construct positive sample patches p containing different noise characteristics in the same infrared image. i With negative sample patch n i The network is guided to bring structurally consistent regions closer together and distance noise-dominant regions from each other in the feature space, thereby suppressing non-Gaussian noise caused by detector inconsistencies and thermal noise during infrared imaging; SSIM is the structural similarity index, I out To output the enhanced result, Φ(•) represents the output features of the pre-trained VGG-19 network conv3_3 layer, s(•,•) represents the cosine similarity, τ represents the temperature parameter, and f i This is the feature representation of the i-th local region of the image extracted by encoder E; Step 4: Image enhancement Input the infrared image to be enhanced into the trained network, and output the enhancement result I. out The output pixel values are limited to the range of 0 and 1 by using the tanh function.
2. The deep learning-based unpaired infrared image enhancement method according to claim 1, characterized in that: Each convolutional layer in the encoder E is followed by batch normalization and LeakyReLU activation to stabilize training and prevent gradient vanishing.
3. The deep learning-based unpaired infrared image enhancement method according to claim 1, characterized in that: The residual module R contains 3 to 5 dense residual blocks, 3 to 5 channel residual blocks, and 3 to 5 deformable convolutional residual blocks, with a total of no less than 9.
4. The deep learning-based unpaired infrared image enhancement method according to claim 1, characterized in that: The perceived loss L per The weights λ² and the self-supervised reconstruction loss L rec Weight λ1 and noise contrast loss L con The weights λ3 are set in a ratio of λ1:λ2:λ3 = 2:1:0.5 to balance brightness restoration and detail enhancement. The self-supervised reconstruction loss L... rec As the main loss term, it is used to ensure the stability of the overall structure and brightness distribution of the images before and after enhancement; perceptual loss L per Used to supplement high-level semantic and detailed information; noise contrast loss L con As an auxiliary constraint, it can effectively suppress non-Gaussian noise in infrared imaging with a small weight, thereby achieving a balance between detail enhancement and noise suppression while ensuring structural consistency.
5. The deep learning-based unpaired infrared image enhancement method according to claim 1, characterized in that: The convolution output of the deformable convolutional residual block satisfies: ， In the formula, ω k p is the convolution weight, p0 is the reference coordinate corresponding to the current convolution position in the output feature map, p k The coordinates of the standard convolution sampling points, Δp k This is a learnable offset.