Infrared image texture information recovery method based on thermal radiation imaging principle

Abstract

The invention relates to an infrared image texture information recovery method based on a thermal radiation imaging principle, and belongs to the field of computer image processing methods. The method comprises the following steps: firstly, according to a Stefan-Boltzmann thermal radiation law, rebuilding a thermal radiation field image in combination with temperature information and emissivity of different substances in the image; in the reconstruction process, a Gabor filter is used for denoising so as to enhance the image quality; then, referring to a Retinex theory, and respectively using a GAN (Generative Adversarial Network) and a U-Net network to obtain two parts of texture information and redundant information from the thermal radiation field image so as to perform infrared texture information recovery; and finally, performing loss calculation based on two loss functions of adversarial loss and pixel loss so as to ensure a texture recovery effect. Based on the thermal radiation principle, the infrared image texture recovery is guided through the visible light, the potential texture information of the infrared image is deeply mined, and the visual effect of the infrared image is remarkably improved.

Description

Technical Field

[0001] The invention relates to the technical field of image processing, and in particular to an infrared image texture information restoration method based on the thermal radiation imaging principle. Background Art

[0002] Infrared imaging technology can provide stable imaging effects all day long under low-light conditions such as night, rain, and fog, so it is widely used in military, search and rescue, industry, agriculture, wildlife protection, medical diagnosis and other fields. However, due to the limitations of thermal radiation imaging technology, ordinary medium- and long-wave infrared images often lack texture information. Although short-wave infrared images can provide clear details and contours, due to their short wavelengths, they are easily affected by environmental factors such as water vapor and dust when propagating in the atmosphere, and the cost is relatively high. Therefore, the current medium- and long-wave infrared imaging devices face the challenge of lack of texture information, and there is an urgent need for a low-cost infrared imaging technology with clear texture. It is difficult to improve image quality from the hardware aspect, and restoring texture details through infrared image enhancement algorithms is a feasible solution.

[0003] At present, conventional infrared image enhancement technologies, such as those based on Generative Adversarial Network (GAN) and Transformer, can only enhance existing texture details, and it is difficult to add or mine new texture information. In the past two years, some scholars have proposed methods to use visible light to guide infrared image enhancement. These methods further increase texture details on the basis of the original infrared image by fusing visible light information. However, this method requires data from both infrared and visible light modalities, and its practical value is not high. Existing infrared enhancement technologies rarely conduct in-depth analysis from the perspective of thermal imaging principles, and it is difficult to effectively mine the potential texture information of infrared images. Therefore, it is urgent to develop a new infrared texture information restoration technology to solve the above problems. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide an infrared texture restoration method guided by visible light based on thermal radiation imaging theory to solve the problem of insufficient texture information in medium and long wave infrared imaging technologies.

[0005] The technical solution of the present invention relates to an infrared texture information restoration method based on the principle of thermal radiation imaging. Specifically, the thermal radiation field image is first reconstructed according to the Stefan-Boltzmann thermal radiation law, and then the encoding method of texture details in the visible light image is learned by using GAN to restore the potential texture information in the infrared image.

[0006] The specific steps of this technical solution are:

[0007] S1: The thermal radiation field image is reconstructed according to the Stefan-Boltzmann thermal radiation law, wherein during the reconstruction process, a Gabor filter is used for denoising and the emissivity parameters are adjusted to enhance the contrast of the target information.

[0008] S2: A texture restoration model is constructed based on the Retinex theory, and the generative adversarial network GAN and U-Net network are used to obtain texture information and redundant information from the thermal radiation field image to restore infrared texture information.

[0009] S3: During the model training process of the texture restoration model, the total loss function constructed based on the adversarial loss and pixel loss is used to calculate the loss to ensure the texture restoration effect.

[0010] More specifically, the specific steps of reconstructing the thermal radiation field image according to the Stefan-Boltzmann thermal radiation law in S1 are as follows:

[0011] S1.1: Combined with the shooting environment parameters, temperature data is obtained from the infrared data, and the temperature data is processed using a Gabor filter to remove the noise introduced in the thermal radiation transmission and imaging process.

[0012] S1.2: Obtain the material category corresponding to different pixel points by performing semantic segmentation on the infrared image, determine the emissivity corresponding to the material category, and adjust the emissivity value to enhance the contrast of the infrared target.

[0013] S1.3: Reconstruct the infrared radiation field using the Stefan-Boltzmann thermal radiation law, where the Stefan-Boltzmann thermal radiation law is: E = εσT 4 , where E represents the radiation energy per unit area, in watts per square meter (W / m 2 ), ε is the average emissivity of non-black bodies, which is a positive number less than 1, indicating the ratio of the radiation capacity of non-black bodies to that of black bodies. σ is the Stefan-Boltzmann constant, which is 5.67×10 -8 W / (m 2 ·K 4 ). T is absolute temperature, the unit is Kelvin (K).

[0014] S1.4: Use the Gabor filter again to eliminate the noise information generated or amplified during the thermal radiation field image reconstruction process.

[0015] Specifically, in the above S2, a texture restoration model is constructed according to the Retinex theory, and the texture information and redundant information are obtained from the thermal radiation field image using the generative adversarial network GAN and the U-Net network respectively, so as to restore the infrared texture information in the following specific steps:

[0016] S2.1: Preprocess the thermal radiation field image and visible light image first:

[0017] S2.1.1: Adjust image contrast by dynamic histogram equalization;

[0018] S2.1.2: Adjust image brightness: Use a visible light image taken under normal lighting conditions as a reference image, and adjust the brightness of the thermal radiation field image and visible light image used for training to match the brightness of the reference image to eliminate the difference in brightness of the visible light image under different lighting conditions.

[0019] S2.1.3: Apply Gabor filter to remove noise in visible light images and enhance the texture details of visible light images.

[0020] S2.2: Input the thermal radiation field image to the first generator G T At the same time, the visible light image is input into the discriminator D.

[0021] The first generator G T The ResNet structure is used, which consists of 9 residual blocks. After training, the first generator G T It enables the thermal radiation field image I to learn the texture encoding method of the visible light image and output the texture restoration image I Tex The discriminator D is built based on PatchGAN and consists of 5 convolutional layers.

[0022] S2.3: Input the thermal radiation field image into the second generator G n Separating redundant information that affects the expression of texture information n . The generator G n The U-Net framework is adopted, which consists of 8 downsampling and 8 upsampling convolutional layers.

[0023] To be more specific, during the model training process, the total loss function includes adversarial loss and pixel loss.

[0024] The adversarial loss is expressed as: Among them, I v is a visible light image, and I is a thermal radiation field image. D(·) represents the judgment of the discriminator on the input, and the output value is between 0 and 1. A value close to 1 represents real data, and a value close to 0 represents generated data. T (I) represents the data generated by the generator. By generating adversarial losses, the generator and discriminator are continuously optimized to improve the model training effect.

[0025] The pixel loss is expressed as: L pix =||I v -G T(I)||1. ||·||1 represents the L1 norm. The pixel loss is used to constrain the infrared texture restoration image to be consistent with the visible light image at the pixel level.

[0026] The total loss function is expressed as: L = λ1L G +λ2L pix λ1 represents the weight parameter of adversarial loss, and λ2 represents the weight parameter of pixel loss. The training effect of the model is guaranteed by combining adversarial loss and pixel loss.

[0027] The beneficial effects of the present invention are as follows: (1) The present invention deeply understands the principle of thermal radiation imaging, reconstructs the radiation field based on the Stefan-Boltzmann method, effectively eliminates the noise introduced in the infrared imaging process through the Gabor filter, and enhances the contrast of the infrared target by adjusting the ε value; (2) The present invention guides infrared texture encoding through visible light, effectively restoring the potential texture information; (3) The present invention learns the texture expression of visible light through GAN instead of directly fusing the visible light texture features, which can be used to achieve low-cost and clear-texture medium and long-wave infrared imaging. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flow chart of the method for restoring texture information of infrared images in the present invention;

[0029] Figure 2 It is the visualization result of the infrared image texture information restoration method of the present invention and other comparative methods. DETAILED DESCRIPTION

[0030] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments, but the protection scope of the present invention is not limited to the described contents.

[0031] like Figure 1 As shown, a method for restoring infrared image texture information based on the principle of thermal radiation imaging, the specific steps are:

[0032] S1: The thermal radiation field image is reconstructed according to the Stefan-Boltzmann thermal radiation law, wherein during the reconstruction process, a Gabor filter is used for denoising and the emissivity parameters are adjusted to enhance the contrast of the target information.

[0033] S1 specifically includes the following steps:

[0034] S1.1: Extract temperature data from infrared images and process the temperature data using Gabor filters to remove noise introduced during thermal radiation transmission and imaging;

[0035] S1.2: Obtain the material category corresponding to different pixel points by performing semantic segmentation on the infrared image, determine the emissivity corresponding to the material category, and adjust the emissivity value to enhance the contrast of the infrared target;

[0036] S1.3: Reconstruct the infrared radiation field based on the Stefan-Boltzmann law of thermal radiation.

[0037] The Stefan-Boltzmann law of thermal radiation is expressed as: E = εσT 4 , E represents the radiation energy per unit area, the unit is watt per square meter W / m 2 , ε is the average emissivity of non-black bodies, which indicates the ratio of the radiation capacity of non-black bodies to that of black bodies; σ is the Stefan-Boltzmann constant, which is 5.67×10-8W / (m 2 ·K 4 ), T is the absolute temperature, the unit is Kelvin (K).

[0038] S1.4: Use the Gabor filter again to eliminate the noise information generated or amplified during the thermal radiation field image reconstruction process.

[0039] S2: According to the Retinex theory, a texture restoration model is constructed, and the generative adversarial network GAN and U-Net network are used to obtain texture information and redundant information from the thermal radiation field image to restore infrared texture information.

[0040] S2 specifically includes the following steps:

[0041] S2.1: First, perform the following three types of preprocessing operations on the thermal radiation field image and the visible light image:

[0042] S2.1.1: Adjust image contrast by dynamic histogram equalization;

[0043] S2.1.2: Adjust image brightness: Use a visible light image under normal lighting as a reference image, and adjust the brightness of the radiation field image and visible light image used for training to match the brightness of the reference image to eliminate the brightness difference of the visible light image under different lighting conditions;

[0044] S2.1.3: Use Gabor filter to eliminate the noise information in the visible light image and enhance the texture details of the visible light image;

[0045] S2.2: Input the thermal radiation field image to the first generator G T At the same time, the visible light image is input into the discriminator D, and the first generator G T The ResNet structure is used, which consists of 9 residual blocks. After training, the thermal radiation field image can learn the texture encoding method of the visible light image and output the texture recovery image.

[0046] S2.3: Input the thermal radiation field image into the second generator G n Separate the redundant information that affects the expression of texture information, where the second generator G n The U-Net framework is adopted, which consists of 8 downsampling and 8 upsampling convolutional layers.

[0047] S3: During the model training process of the texture restoration model, the total loss function constructed based on the adversarial loss and pixel loss is used to calculate the loss to ensure the texture restoration effect.

[0048] During the model training process, two different loss functions were designed, namely adversarial loss and pixel loss, considering the specific requirements of the task.

[0049] Among them, the adversarial loss is expressed as Among them, I v is the visible light image, and I is the thermal radiation field image. The pixel loss is expressed as: L pix =||I v -G T (I)||1. The total loss function is expressed as: L = λ1L G +λ2L pix , where λ1=0.016, λ2=0.002.

[0050] According to the loss function calculation results, the generator (G T , G n ) and the network parameters of the discriminator D to make the training model converge.

[0051] This example is implemented based on the pytorch framework. The initial learning rate is set to 0.001 and gradually decays after 20 rounds of training until it decays to 0 after 160 epochs. The model training process is completed on a single NVIDIA RTX4090 GPU, while the test is performed on a single NVIDIA RTX4060Ti GPU. 861 pairs of RGB-T images are randomly selected from the public jungle dataset ODinMJ for training.

[0052] The present invention also makes a visual comparison between the experimental results of the proposed method and the classical histogram equalization method. Figure 2As shown. Further, non-reference indicators such as average gradient, entropy enhancement metric, spatial frequency, natural image quality estimation and perception-based image quality estimation are used for quantitative evaluation. Among them, average gradient (AG) and spatial frequency (SF) indicate the clarity of the image. The larger the value, the clearer the image. Entropy enhancement metric (EME) is used to evaluate the performance of the algorithm in contrast improvement. The larger the value, the higher the image contrast. The natural image quality estimation (NIQE) indicator is used to measure the naturalness or authenticity of the image. The smaller the NIQE value, the higher the image quality. The perception-based image quality estimation (PIQE) indicator calculates the quality score of the image through the block structure and noise characteristics of the image. The smaller the PIQE value, the higher the image quality. The evaluation results are shown in Table 1.

[0053] Depend on Figure 2 It can be seen that compared with the histogram equalization method, the method proposed in the present invention restores more texture details in the image and provides better visual effects. As shown in Table 1, for each of the listed evaluation indicators, the method proposed in the present invention is higher than the histogram equalization, indicating that the method proposed in the present invention restores more texture details, higher contrast, better naturalness and perception effects, and is superior to the histogram equalization method in all aspects.

[0054] Table 1: Quantitative evaluation of infrared texture information recovery

[0055]

[0056] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A method for restoring infrared image texture information based on the principle of thermal radiation imaging, characterized in that: The following steps are involved: S1: Reconstruct the thermal radiation field image according to the Stefan-Boltzmann thermal radiation law, wherein during the reconstruction process, a Gabor filter is used for denoising, and the emissivity parameters are adjusted to enhance the contrast of the target information; S2: Construct a texture restoration model based on the Retinex theory, and use the generative adversarial network GAN and the U-Net network to obtain texture information and redundant information from the thermal radiation field image to restore infrared texture information; S3: During the model training process of the texture restoration model, the total loss function constructed based on the adversarial loss and pixel loss is used to calculate the loss to ensure the texture restoration effect.

2. The method for restoring infrared image texture information according to claim 1, characterized in that: S1 specifically includes the following steps: S1.1: Extract temperature data from infrared images and process the temperature data using Gabor filters to remove noise introduced during thermal radiation transmission and imaging; S1.2: Obtain the material category corresponding to different pixel points by performing semantic segmentation on the infrared image, determine the emissivity corresponding to the material category, and adjust the emissivity value to enhance the contrast of the infrared target; S1.3: Reconstruct the infrared radiation field based on the Stefan-Boltzmann thermal radiation law; S1.4: Use the Gabor filter again to eliminate the noise information generated or amplified during the thermal radiation field image reconstruction process.

3. The method for restoring infrared image texture information according to claim 1, characterized in that: S2 specifically includes the following steps: S2.1: First, perform the following three types of preprocessing operations on the thermal radiation field image and the visible light image: S2.1.1: Adjust image contrast by dynamic histogram equalization; S2.1.2: Adjust image brightness: Use a visible light image under normal lighting as a reference image, and adjust the brightness of the radiation field image and visible light image used for training to match the brightness of the reference image to eliminate the brightness difference of the visible light image under different lighting conditions; S2.1.3: Use Gabor filter to eliminate the noise information in the visible light image and enhance the texture details of the visible light image; S2.2: Input the thermal radiation field image to the first generator G T At the same time, the visible light image is input into the discriminator D, and the first generator G T The ResNet structure is used. After training, the thermal radiation field image can learn the texture encoding method of the visible light image and output the texture restoration image. S2.3: Input the thermal radiation field image into the second generator G n Separate the redundant information that affects the expression of texture information, where the second generator G n Adopt U-Net framework.

4. The method for restoring infrared image texture information according to claim 1, characterized in that: The adversarial loss is expressed as: I v is the visible light image, I is the thermal radiation field image; the pixel loss is expressed as: L pix =||I v -G T (I)||1; the total loss function is expressed as: L = λ1L G +λ2L pix .

Images