Infrared image enhancement method based on thermal signal decomposition

Abstract

The invention relates to an infrared image enhancement method based on thermal signal decomposition, and belongs to the technical field of computer image processing. The method comprises the steps that S1, according to the Stefan-Boltzmann law, a thermal radiation field image I 'D is reconstructed through temperature information T, a direct thermal emission component ID without texture information is modeled, and an original infrared image I is decomposed into the direct thermal emission component ID and an environment reflection component IE based on a signal source; s2, subtracting the thermal radiation field image I 'D from the original infrared image I, separating out a part I' E containing texture information, and modeling an environment reflection component IE; s3, potential texture information in the part I 'E containing the texture information is deeply mined through an infrared-texture mapping model, enhanced texture information I' ET is obtained, and redundant information I 'En influencing infrared texture information expression is extracted from the part I' E containing the texture information through a redundant information separation network Gn; and S4, fusing the thermal radiation field image I 'D and the enhanced texture information I' ET to obtain an enhanced infrared image I '.

Description

Technical Field

[0001] The invention relates to the technical field of image processing, and in particular to an infrared image enhancement method based on thermal signal decomposition. Background Art

[0002] Infrared imaging technology plays an important role in many key fields such as military reconnaissance, security monitoring and industrial inspection. Since infrared images often lack texture information and have low contrast, they face challenges in target recognition and image analysis. Infrared image enhancement technology significantly improves these problems by improving the clarity of image features. Infrared image enhancement technology is mainly divided into two categories: traditional methods and deep learning-based methods. Traditional infrared image enhancement methods are mainly divided into spatial domain processing and frequency domain processing. Spatial domain image enhancement methods enhance contrast by redistributing the pixel values ​​of the image, including techniques such as histogram equalization and contrast stretching. Spatial domain image enhancement methods have the advantages of being intuitive, easy to operate and real-time, and are particularly suitable for processing local areas of the image. Spatial domain image enhancement methods are difficult to consider the overall contrast and details of the image. Frequency domain image enhancement methods enhance images by transforming the image from the spatial domain to the frequency domain and applying frequency domain filtering. Finally, the image is restored to the spatial domain through inverse Fourier transform. This method has excellent denoising capabilities and precise control of different frequency components, but this method has limitations in processing local details and may also cause frequency aliasing.

[0003] With the rapid development of deep learning technology, infrared image enhancement methods based on deep learning have shown excellent performance. Researchers have effectively improved the contrast between the target and the background by using convolutional neural networks (CNN) to learn filters to predict the characteristics of the target and the background, making the target more prominent in the image. In addition, in order to further improve the quality of infrared images, some scholars have introduced generative adversarial networks (GANs). This technology not only enhances the texture and edge details of the image through adversarial training, but also improves the overall visual effect of the image. However, current infrared image enhancement methods mainly focus on denoising, contrast enhancement, and recombination of texture information. These operations do not essentially introduce new information into the image, thus limiting the further improvement of the enhancement effect. Therefore, how to effectively mine and utilize the potential texture information in infrared images has become a key issue that needs to be solved urgently. Summary of the invention

[0004] In order to solve the above technical problems and effectively mine and utilize the potential texture information in infrared images, this application proposes an infrared image enhancement method based on thermal signal decomposition. Specifically, according to the source of the thermal signal, the infrared image I is decomposed into a direct thermal emission component I D (no texture) and ambient reflection component I EThe specific scheme is as follows: S1. According to the Stefan-Boltzmann thermal radiation law, the thermal radiation field image I' is reconstructed from the temperature information T D , modeling the direct emission component I D , the original infrared image I is decomposed into the direct thermal emission component I D and the ambient reflection component I E Two parts; S2, subtract the reconstructed thermal radiation field image I' from the original infrared image I D , separate the part containing texture information I' E , modeling the ambient reflection component I E ; S3, design infrared-texture mapping model, through visible light image I v Digging deeper into the part containing texture information I' E The potential texture information in the image is obtained to obtain the enhanced texture information I' ET , further separated by redundant information network G n From the part containing texture information I' E Extract the redundant information I' that affects the expression of infrared texture information En ; S4, the thermal radiation field image I ' D And the enhanced texture information I' ET The enhanced infrared image I' is obtained by fusion. It should be understood that the original infrared image I can be decomposed into direct thermal emission components I according to the source of the thermal signal. D (no texture) and ambient reflection component I E (Carrying Texture) Two parts.

[0005] Further optionally, in S4, redundant information I' is extracted En Thermal radiation field image I' D And the enhanced texture information I' En Perform fusion, including: S4.1, according to the thermal radiation field image I' D And the enhanced texture information I' ET Calculate adaptive weights

[0006] Weight w, which is expressed as: Among them, w PIQE represents the PIQE weight component, w AG represents the AG weight component, w NIQE represents the NIQE weight component, w STD Represents the STD weight component; S4.2, using the adaptive weight w to fuse to obtain the enhanced infrared image I', I'=I' D ·w+I' ET ·(1-w).

[0007] Further optional, wPIQE 、w AG 、w NIQE and w STD The expressions are as follows:

[0008]

[0009] Optionally, the infrared-texture mapping model in S3 is generated by the generator G T and discriminator D; generator G T The ResNet structure is adopted, which consists of 9 residual blocks; the discriminator D is based on PatchGAN and consists of 5 convolutional layers; the redundant information separation network in S3 adopts the U-Net framework, which consists of 8 downsampling and 8 upsampling convolutional layers.

[0010] Further optionally, during the training process of the infrared-texture mapping model in S3, a total loss function including adversarial loss, perceptual loss, pixel loss, reconstruction loss and style loss is used for loss calculation; the adversarial loss is expressed as: The perceptual loss can be expressed as: Among them, W i,j represents the width of the i-th layer and j-th feature map, H i,j represents the height of the i-th layer and j-th feature map, φ i,j represents the function of the i-th layer and j-th feature map extracted from the VGG network; the pixel loss is expressed as: L pix =||I v -G T (I' E )||1; the reconstruction loss is expressed as: L rec =||I' E -I" E ||1, where I” E =I' ET ×I' En , represents the enhanced texture information I' ET With redundant information I' En Reconstructed pseudo thermal radiation image; the style loss is expressed as: L style =||I v -G T (I v )||1; the total loss function is expressed as:

[0011] L=λ1L G +λ2L VGG / i.j +λ3L pix +λ4L rec +λ5L style .

[0012] The beneficial effects of the present invention are: (1) The present invention divides the original infrared image into a direct emission component and an ambient emission component according to the signal source, and performs physical modeling and processing respectively, thereby improving the effects of denoising and texture enhancement. (2) The present invention constructs a texture-redundant information decomposition model based on the infrared image texture degradation mechanism, and effectively separates and extracts infrared texture information components. (3) The present invention introduces adaptive fusion, which automatically adjusts the image weights based on the reference-free evaluation index, efficiently fuses the texture features of different images, and significantly improves the image reconstruction effect. (4) The present invention uses a total loss function including adversarial loss, perceptual loss, pixel loss, reconstruction loss and style loss for loss calculation, which effectively restores the texture details of the infrared image. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 A schematic diagram of a flow chart of an infrared image enhancement method based on thermal signal decomposition in the present invention;

[0014] Figure 2 It is the visualization result of image enhancement by the infrared image enhancement method based on thermal signal decomposition in the present invention and other contrast methods. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0016] See also Figure 1 The infrared image enhancement method based on thermal signal decomposition in the present invention comprises:

[0017] S1. According to the Stefan-Boltzmann law, the thermal radiation field image I' is reconstructed from the temperature information T D , modeling the direct thermal emission component I without texture information D , the original infrared image I is decomposed into direct thermal emission component I based on the signal source D and the ambient reflection component I E Two parts.

[0018] S2. Subtract the thermal radiation field image I' from the original infrared image I D , separate the part containing texture information I' E , modeling the ambient reflection component I E .

[0019] S3. Design infrared-texture mapping model, through visible light image Iv Digging deeper into the part containing texture information I' E The potential texture information in the image is obtained to obtain the enhanced texture information I' ET , through redundant information separation network G n From the part containing texture information I' E Extract the redundant information I' that affects the expression of infrared texture information En .

[0020] For S3, the infrared-texture mapping model is generated by the generator G T and discriminator D; generator G T The ResNet structure is adopted, which consists of 9 residual blocks; the discriminator D is based on PatchGAN and consists of 5 convolutional layers; the redundant information separation network adopts the U-Net framework, which consists of 8 downsampling and 8 upsampling convolutional layers.

[0021] S4, the thermal radiation field image I' D And the enhanced texture information I' ET The fusion is performed to obtain the enhanced infrared image I'.

[0022] Specifically, in S4, the thermal radiation field image I' D And the enhanced texture information I' ET Perform fusion, including: S4.1, according to the thermal radiation field image I' D And the enhanced texture information I' ET Calculate the adaptive weight w, which is expressed as: Among them, w PIQE represents the PIQE weight component, w AG represents the AG weight component, w NIQE represents the NIQE weight component, w STD Represents the STD weight component; S4.2, using the adaptive weight w to fuse to obtain the enhanced infrared image I', I'=I' D ·w+I' ET ·(1-w).

[0023] The above PIQE 、w AG 、w NIQE and w STD The expressions are as follows:

[0024]

[0025] In the present invention, the adaptive weight is calculated by four parts: perception based Image Quality Evaluator (PIQE), natural image quality evaluator (NIQE), average gradient (AG) and standard deviation (STD).

[0026] In addition, during the training process of the infrared-texture mapping model, the total loss function including adversarial loss, perceptual loss, pixel loss, reconstruction loss and style loss is used for loss calculation; the adversarial loss is expressed as: The perceptual loss can be expressed as: Among them, W i,j represents the width of the i-th layer and j-th feature map, H i,j represents the height of the i-th layer and j-th feature map, φ i,j represents the function of the i-th layer and j-th feature map extracted from the VGG network; the pixel loss is expressed as: L pix =||I v -G T (I' E )||1; the reconstruction loss is expressed as: L rec =||I' E -I" E ||1, where I” E =I' ET ×I' En , represents the enhanced texture information I' ET With redundant information I' En Reconstructed pseudo thermal radiation image; the style loss is expressed as: L style =||I v -G T (I v )||1; the total loss function is expressed as: L = λ1L G +λ2L VGG / i.j +λ3L pix +λ4L rec +λ5L style .

[0027] Specifically, the perceptual loss is used to ensure that the deep texture encoding method of the network output image is closer to the visible light image. In the present invention, the perceptual loss is defined based on the pre-trained 19-layer Visual Geometry Group (VGG) network. The pixel loss is used to constrain the output image to be closer to the visible light image. vMaintain consistency at the pixel level. Use reconstruction loss to constrain the output data to maintain the original features of the infrared image. Use style loss to constrain the input and output data of the model to express consistent styles.

[0028] In addition, the embodiment of the present invention is implemented based on the pytorch framework, the initial learning rate is set to 0.001, and it gradually decays after 20 rounds of training until it decays to 0 after 160 epochs. The model training process is completed on a single NVIDIARTX4090 GPU, and the test is performed on a single NVIDIARTX4060 Ti GPU. 861 pairs of RGB-T images are randomly selected from the public jungle dataset ODinMJ for training.

[0029] This patent also makes a visual comparison between the experimental results of the proposed method and the classic histogram equalization method. Figure 2 As shown. Further, non-reference indicators such as Average Gradient (AG), Measure of Enhancement by Entropy (EME), Spatial Frequency (SF) and Perception based Image Quality Evaluator (PIQE) are used for quantitative evaluation. Among them, AG and SF represent the clarity of the image. The larger the value, the clearer the image. EME is used to evaluate the performance of the algorithm in contrast improvement. The larger the value, the higher the image contrast. The PIQE indicator calculates the quality score of the image through the block structure and noise characteristics of the image. The smaller the PIQE value, the higher the image quality. The evaluation results are shown in Table 1.

[0030] Depend on Figure 2 It can be seen that compared with the histogram equalization method, the method proposed in this patent shows a better enhancement effect. As shown in Table 1, for each of the evaluation indicators listed, the method proposed in this patent is higher than the histogram equalization, indicating that the method proposed in this patent restores more texture details and better perceptual effects, and is superior to the histogram equalization method in all aspects.

[0031] Table 1 Quantitative evaluation of infrared texture information recovery

[0032]

[0033] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. An infrared image enhancement method based on thermal signal decomposition, characterized in that: The following steps are involved: S1. According to the Stefan-Boltzmann law, the thermal radiation field image I' is reconstructed from the temperature information T D , modeling the direct thermal emission component I without texture information D , the original infrared image I is decomposed into direct thermal emission component I based on the signal source D and the ambient reflection component I E Two parts; S2. Subtract the thermal radiation field image I' from the original infrared image I D , separate the part containing texture information I' E , modeling the ambient reflection component I E ; S3. Design infrared-texture mapping model, through visible light image I v Digging deeper into the part containing texture information I' E The potential texture information in the image is obtained to obtain the enhanced texture information I' ET , through redundant information separation network G n From the part containing texture information I' E Extract the redundant information I' that affects the expression of infrared texture information En ; S4, the thermal radiation field image I' D And the enhanced texture information I' ET The fusion is performed to obtain the enhanced infrared image I'.

2. The infrared image enhancement method according to claim 1, characterized in that: In S4, the thermal radiation field image I' D And the enhanced texture information I' ET Integration, including: S4.

1. Based on the thermal radiation field image I' D And the enhanced texture information I' ET Calculate the adaptive weight w, which is expressed as: Among them, w PIQE represents the PIQE weight component, w AG represents the AG weight component, w NIQE represents the NIQE weight component, w STD represents the STD weight component; S4.

2. Use adaptive weight w to fuse and obtain enhanced infrared image I', I'=I' D ·w+I' ET ·(1-w).

3. The infrared image enhancement method according to claim 2, characterized in that: w PIQE 、w AG 、w NIQE and w STD The expressions are as follows:

4. The infrared image enhancement method according to claim 1, characterized in that: The S3 mid-infrared-texture mapping model is composed of the generator G T and discriminator D; generator G T The ResNet structure is used, which consists of 9 residual blocks; the discriminator D is based on PatchGAN and consists of 5 convolutional layers; The redundant information separation network in S3 adopts the U-Net framework, which consists of 8 downsampling and 8 upsampling convolutional layers.

5. The infrared image enhancement method according to claim 1, characterized in that: During the training of the infrared-texture mapping model in S3, the total loss function including adversarial loss, perceptual loss, pixel loss, reconstruction loss and style loss is used for loss calculation; The adversarial loss is expressed as: The perceptual loss can be expressed as: Among them, W i,j represents the width of the i-th layer and j-th feature map, H i,j represents the height of the i-th layer and j-th feature map, φ i,j represents the function of the i-th layer and j-th feature map extracted from the VGG network; The pixel loss is expressed as: L pix =||I v -G T (I' E )||1; The reconstruction loss is expressed as: L rec =||I' E -I″ E ||1, where I″ E =I' ET ×I' En , represents the enhanced texture information I' ET With redundant information I' En Reconstructed pseudo thermal radiation image; The style loss is expressed as: L style =||I v -G T (I v )||1; The total loss function is expressed as: L = λ1L G +λ2L VGG / i.j +λ3L pix +λ4L rec +λ5L style .

Images