A thermal infrared image enhancement method based on global and local feature fusion modulation

Abstract

The application discloses a thermal infrared image enhancement method based on global and local feature fusion modulation, which aims at improving the problems of poor contrast and fuzzy details of the original thermal infrared image and improving the visual quality of the thermal infrared image. The method is realized based on a thermal infrared image enhancement model, and the thermal infrared image enhancement model is composed of a feature extraction module and an image enhancement backbone module. The feature extraction module is used for processing the low-quality infrared image to generate a feature tensor. The image enhancement backbone module utilizes the global and local feature modulation parameters obtained by the feature tensor transformation to perform global and local fusion modulation on the feature map of the low-quality infrared image. The method can effectively utilize the global and local information, improve the overall image contrast, ensure the local image detail texture definition, and realize high-quality thermal infrared image enhancement.

Description

Technical Field

[0001] This invention relates to the field of image processing, specifically to a thermal infrared image enhancement method utilizing global and local feature fusion modulation. Background Technology

[0002] Thermal infrared image enhancement is a crucial technology in thermal imaging, primarily used to improve the quality and visual effects of thermal infrared images. Due to inherent limitations in thermal imaging principles, the complexity of atmospheric transmission environments, and the sensitivity of infrared detectors, the raw images produced by infrared imaging systems often fall short of ideal results, mainly exhibiting low signal-to-noise ratios, poor contrast, and blurred edge details. To address these issues, the most economical and effective approach is to conduct targeted research on infrared image quality enhancement algorithms from a digital image processing perspective, thereby significantly and effectively improving the imaging performance of infrared imaging systems.

[0003] Thermal infrared image enhancement helps us better utilize thermal infrared images, achieving better results in various application scenarios. In the field of intelligent driving, thermal infrared images can help detect road obstacles in adverse weather conditions. Image enhancement provides a clearer view of road conditions, leading to safer driving. In the field of security monitoring, thermal infrared images can help discover potential threats in dark environments. Image enhancement provides a clearer view of targets, enabling more effective monitoring. In the medical field, thermal infrared images can help detect abnormal temperature distributions in the human body, enabling early detection of some diseases. Image enhancement provides a clearer view of temperature distribution, leading to more accurate diagnosis.

[0004] Histogram-based enhancement algorithms are a widely studied category in thermal infrared image enhancement research. These algorithms are primarily developed from histogram equalization (HE) algorithms, representing improvements on the classic HE algorithm. This is because the classic HE algorithm is prone to over-amplifying background noise and oversaturating local bright areas when enhancing infrared image contrast. For example, the paper "A real-time contrast enhancement algorithm for infrared images based on plateau histogram" effectively improves the algorithm's adaptability to different scenes by selecting a median value from the set of local maxima of the histogram as an adaptive platform threshold. The paper "Infrared contrast enhancement through log-power histogram modification" uses a log-power function transformation to modify the histogram to prevent excessive background and noise enhancement and to improve detail information. However, these algorithms rely solely on histogram information and cannot completely prevent over- or under-enhancement, while also being sensitive to noise and other interference factors in the image.

[0005] With the development of deep learning technology, deep neural networks, thanks to their superior feature extraction capabilities, are increasingly being applied to thermal infrared image enhancement tasks. For example, the paper "Single infrared image enhancement using deep convolutional neural network" proposes a deep learning method for single-frame infrared image enhancement, using a fully convolutional neural network to generate images with enhanced contrast and detail. It also incorporates a conditional generative adversarial network into the optimization framework to avoid amplifying background noise, further enhancing contrast and detail. The paper "Target attention deep neural network for infrared image enhancement" proposes an infrared image enhancement method based on a target attention mechanism. It constructs a joint convolutional unit to comprehensively mine complementary multi-scale spatial features, and designs a target attention unit to further refine features, achieving end-to-end image enhancement. While the above deep learning methods significantly outperform traditional methods in enhancing infrared image contrast, these methods inevitably produce artifacts in detail edge areas, resulting in insufficient detail clarity. Summary of the Invention

[0006] This invention discloses a thermal infrared image enhancement method based on the fusion modulation of global and local features. The main goal of this method is to address common problems in raw thermal infrared images, such as low contrast and blurred details, thereby improving the visual quality of thermal infrared images. This invention processes raw thermal infrared images through the fusion modulation of global and local features. Global features mainly reflect the overall information of the image, such as overall brightness and contrast; local features mainly reflect the local information of the image, such as edges and textures. By fusing global and local features, the characteristics of the image can be described more comprehensively and accurately. Then, by modulating global and local features, the contrast of the raw thermal infrared image can be improved, achieving an ideal balance between brightness and contrast. Simultaneously, image details can be enhanced, making local information such as texture and edges clearer. Finally, the visual quality of the thermal infrared image processed by this invention is significantly improved. Both overall and local contrast are improved, and details become clearer. This not only enhances the observer's perception of the image but also provides a better foundation for subsequent image processing and analysis.

[0007] The specific implementation steps of the method of the present invention are as follows:

[0008] A thermal infrared image enhancement method based on global and local feature fusion modulation is proposed. A low-quality thermal infrared image is input into a thermal infrared image enhancement model. Through the collaborative efforts of a feature extraction module and an image enhancement backbone module, the global and local information of the image is fully utilized to enhance it, outputting a thermal infrared image with good visual effects. The method for obtaining the thermal infrared image enhancement model is as follows:

[0009] S1: Dataset Construction. Construct a pseudo-infrared image dataset with high-to-low quality pairings. This dataset contains a large number of high-quality ground truth image pairs and low-quality input image pairs for training the model.

[0010] S2: Construction of thermal infrared image enhancement model.

[0011] S21: First, a feature extraction module is constructed, which processes the low-quality pseudo-infrared image to generate a feature tensor.

[0012] S22: Construct an image enhancement backbone module. The image enhancement backbone module uses the global and local feature modulation parameters obtained by the feature tensor transformation to perform global and local fusion modulation on the feature map of the low-quality pseudo-infrared image based on the modulation parameters of the global and local features, and outputs it as the model.

[0013] S23: Using high-quality pseudo-infrared images as ground truth, and based on the structural similarity loss function, supervise the model output of step S22, train the feature extraction module and the image enhancement backbone module until convergence, and obtain the thermal infrared image enhancement model.

[0014] In step S1, based on the existing visible light high- and low-quality paired dataset, a large number of high- and low-quality paired pseudo-infrared images are simulated and generated for model training. Specifically:

[0015] First, the images in the visible light high- and low-quality paired dataset are converted into grayscale images. Grayscale images contain only brightness information, which can effectively simulate the characteristics of thermal infrared images. Then, the pixel values ​​of the grayscale images are further adjusted to make them closer to real thermal infrared images, resulting in preliminary pseudo-infrared images.

[0016] Next, for a preliminary set of high- and low-quality paired pseudo-infrared images, various methods are used to further enhance the contrast of the high-quality images. The image with the highest contrast is selected as the high-quality pseudo-infrared image and used as the ground truth for model training. Simultaneously, random contrast compression is used to degrade the low-quality images, simulating various degradation scenarios that infrared images might encounter in practical applications. Images with different degrees of degradation are used as low-quality pseudo-infrared images and as input for model training. Each low-quality pseudo-infrared image is paired with a corresponding high-quality pseudo-infrared image, resulting in multiple pairs of high- and low-quality paired pseudo-infrared images.

[0017] Step S2, the construction of the thermal infrared image enhancement model, is divided into three parts: first, constructing the feature extraction module; second, constructing the image enhancement backbone module; and third, module training. Details are as follows:

[0018] The feature extraction module is divided into two parts: a global feature extraction module and a local feature extraction module. The global feature extraction module works similarly to an encoder; its main task is to extract global information from the image, including contrast and brightness. This information reflects the overall state of the image and plays a crucial role in understanding its global characteristics. The global feature extraction module performs a series of calculations and processing on the input image to ultimately generate a global feature vector. The local feature extraction module, on the other hand, is responsible for extracting local information from the image. This information mainly includes local structural information and contrast information, reflecting the image's details. The local feature extraction module performs a series of calculations and processing on the input image to ultimately generate a three-dimensional matrix of local features. Specifically:

[0019] The global feature extraction module consists of three 32-channel convolutional layers and a global average pooling layer, with each convolutional layer followed by a ReLU activation function. First, the input image is fed into the first convolutional layer, which has a 7×7 kernel and a stride of 2 to increase the receptive field. Then, the output of the first convolutional layer is fed into the second and third convolutional layers, both with a 3×3 kernel and a stride of 2. Finally, the output of the last convolutional layer is fed into the global average pooling layer to reduce the dimensionality of the features while preserving important information, ultimately yielding a global feature vector containing global brightness and contrast information of the input image.

[0020] The local feature extraction module comprises three 32-channel convolutional layers, each followed by a ReLU activation function. These three convolutional layers are cascaded together, with the output of each layer serving as the input to the next. The first convolutional layer has a 7×7 kernel with a stride of 1, while the second and third layers have 3×3 kernels with a stride of 1. Through the sequential operation of these three convolutional layers, rich local structure and contrast information can be extracted from the original image and encoded into a three-dimensional local feature matrix.

[0021] The image enhancement backbone module fully utilizes global and local information to enhance the overall contrast of the image while preserving local details and textures. This module employs a fully convolutional structure, which exhibits excellent translation invariance, ensuring consistent processing results across all pixels. Each pixel in the image is processed through a global feature modulation layer and a local feature modulation layer, achieving comprehensive utilization of both global and local information. In the global feature modulation layer, the generated global feature vector improves the overall visual effect, enhancing contrast and brightness. In the local feature modulation layer, the generated local feature 3D matrix enhances image detail and addresses issues such as local blurring and excessive contrast. Specifically:

[0022] The image enhancement backbone module consists of three convolutional layers with 64 channels each and two ReLU activation functions. In each convolutional layer, all kernels are 1×1 in size with a stride of 1, processing each pixel in the image independently. Following each convolutional layer are a global feature modulation layer and a local feature modulation layer, which work by using scaling and offset operations to modulate the intermediate features of the convolutional layer output. The global feature modulation layer can be modeled as GFM(x ij ) = s g *x ij +b g , where x ij It is the eigenvalue at coordinate position (i,j), s g and bg These are the global feature modulation parameters obtained from the transformation of the global feature vector, namely the global scaling factor and the offset factor. After passing through the global feature modulation layer, the features are fed into the local feature modulation layer, which can be modeled as... Where y ij It is x ij The result after processing by the global feature modulation layer and These are the local feature modulation parameters obtained by transforming the local feature three-dimensional matrix, representing the scaling factor and offset factor of the feature value at the (i,j) coordinate position, respectively.

[0023] Based on the structural similarity loss function, the feature extraction module and the image enhancement backbone module are trained to make low-quality images, after enhancement, as close as possible to the ground truth of high-quality images in terms of brightness, contrast, and structure. Specifically:

[0024] The structural similarity loss function is used as the loss function during the training of the feature extraction module and the image enhancement backbone module. The loss function is defined as follows: Where x is the enhanced image result, y is the high-quality image used as supervision, and μ x It is the average value of x. It is the variance of x, μ y It is the average value of y. σ is the variance of y. xy Let x be the covariance of x and y, and c1 and c2 be two constants used to maintain stability and avoid division by zero. This loss function measures the structural similarity between the enhanced image and the high-quality image. By minimizing this loss function, the enhanced image can be made as close as possible to the high-quality image in terms of brightness, contrast, and structure.

[0025] When the aforementioned thermal infrared image enhancement model is deployed into a thermal infrared imaging system, the system can output visually appealing thermal infrared images in real time. Due to the ingenious design of the feature extraction module and the image enhancement backbone module, the overall number of parameters and computational load of the model are effectively controlled to a very low level. Although the method of this invention involves complex global and local feature extraction and modulation processes, it remains lightweight in terms of parameters and computational load due to the above design. This allows the method of this invention to meet the stringent real-time requirements of thermal infrared imaging systems and to operate smoothly even in resource-constrained environments. Attached Figure Description

[0026] Figure 1 This is a flowchart of the method of the present invention.

[0027] Figure 2This is a schematic diagram of the modulation of the global feature modulation layer and the local feature modulation layer in this invention.

[0028] Figure 3 This is a flowchart of the module training phase of the present invention.

[0029] Figure 4 This is the final enhanced thermal infrared image obtained during the model inference stage of this invention. Detailed Implementation

[0030] The present invention will be further described below with reference to specific embodiments and accompanying drawings. Example

[0031] Based on existing high- and low-quality visible light paired datasets, a large number of high- and low-quality paired pseudo-infrared images are simulated and generated for model training, specifically:

[0032] First, the images in the visible light high- and low-quality paired dataset are converted into grayscale images. Grayscale images contain only brightness information, which can effectively simulate the characteristics of thermal infrared images. Then, the pixel values ​​of the grayscale images are further adjusted to make them closer to real thermal infrared images, resulting in preliminary pseudo-infrared images. Next, for a set of preliminary high- and low-quality paired pseudo-infrared images, various methods are used to further enhance the contrast of the high-quality images. The image with the highest contrast is selected as the high-quality pseudo-infrared image and used as the ground truth for model training. Simultaneously, random contrast compression is used to degrade the low-quality images, which can simulate various degradation situations that infrared images may encounter in practical applications. Images with different degrees of degradation are used as low-quality pseudo-infrared images and used as input for model training. Each low-quality pseudo-infrared image is paired with a corresponding high-quality pseudo-infrared image to obtain multiple pairs of high- and low-quality paired pseudo-infrared images.

[0033] The network module structure diagram proposed by the method of this invention is as follows: Figure 1 As shown, it is mainly divided into a feature extraction module and an image enhancement backbone module.

[0034] The feature extraction module is further divided into a global feature extraction module and a local feature extraction module. The global feature extraction module works similarly to an encoder; its main task is to extract global information from the image, including contrast and brightness. This information reflects the overall state of the image and plays a crucial role in understanding its global characteristics. The global feature extraction module performs a series of calculations and processing on the input image to ultimately generate a global feature vector. The local feature extraction module, on the other hand, is responsible for extracting local information from the image. This information mainly includes local structural information and contrast information, reflecting the image's details. The local feature extraction module performs a series of calculations and processing on the input image to ultimately generate a three-dimensional matrix of local features.

[0035] The process by which the global feature extraction module generates global feature vectors is described in detail below:

[0036] First, the input image is fed into a first convolutional layer with a kernel size of 7×7, a stride of 2, and 32 channels. Then, the output of the first convolutional layer is fed into a second and third convolutional layer, both with a kernel size of 3×3, a stride of 2, and 32 channels. Finally, the output of the third convolutional layer is fed into a global average pooling layer to reduce the dimensionality of the features while preserving important information. This results in a global feature vector of size 1×32, containing global brightness and contrast information of the input image.

[0037] The process by which the local feature extraction module generates a 3D matrix of local features is described in detail below:

[0038] First, the input image is fed into the first convolutional layer, which has a 7×7 kernel, a stride of 1, and 32 channels to help capture large-scale features of the image. Next, two more convolutional layers follow, each with a 3×3 kernel, a stride of 1, and 32 channels, to capture small-scale features. Through the continuous operation of these three convolutional layers, rich local structure and contrast information can be extracted from the original image. Assuming the input image size is H×W, a three-dimensional local feature matrix of size H×W×32 is ultimately generated.

[0039] The image enhancement backbone module utilizes the generated global feature vector and local feature 3D matrix to process each pixel in the image through a global feature modulation layer and a local feature modulation layer, respectively, to improve the overall visual effect of the image and increase its contrast and brightness. Simultaneously, it enhances the image's detail and optimizes issues such as local blurring and excessive contrast. The specific process is shown in

[0040] to

[0042] :

[0040] First, a convolutional layer with a kernel size of 1×1, a stride of 1, and 64 channels is used to process each pixel in the image independently. Then, intermediate features are modulated through a global feature modulation layer and a local feature modulation layer. These layers work by using scaling and offset operations to modulate the intermediate features output from the convolutional layer. Assume the input image size is H×W, and the size of the intermediate features is H×W×64. A fully connected layer transforms the size of the global feature vector and the local feature 3D matrix to 1×64 and H×W×64 respectively, thus obtaining the scaling factor and offset factor (i.e., local feature modulation parameters) corresponding to the 64 channels. Figure 2 As shown, in a single channel, the eigenvalue x at coordinate (i,j) is... ij First, it undergoes processing by a global feature modulation layer, a process that can be modeled as GFM(xij ) = s g *x ij +b g , where s g and b g These are the global scaling factor and offset factor obtained from the global eigenvector transformation. x ij The result after processing by the global feature modulation layer is denoted as y. ij After that, y ij The result z is obtained after processing by the local feature modulation layer. ij This process can be modeled as in and These are the scaling and offset factors (i.e., global feature modulation parameters) of the eigenvalues ​​at coordinates (i,j) obtained by transforming the local feature three-dimensional matrix.

[0041] Repeat the process of segment

[0040] .

[0042] Finally, the intermediate features are processed sequentially through a convolutional layer with a kernel size of 1×1, a stride of 1, and a channel number of 1, a global feature modulation layer, and a local feature modulation layer, resulting in an enhanced result with high overall contrast and high clarity of local details and textures.

[0043] Based on the structural similarity loss function, the feature extraction module and the image enhancement backbone module are trained, such as... Figure 3 As shown.

[0044] The structural similarity loss function is defined as follows:

[0045]

[0046] Where x is the enhanced image result, y is the high-quality image used as supervision, and μ x It is the average value of x. It is the variance of x, μ y It is the average value of y. σ is the variance of y. xy Let x be the covariance of x and y, and c1 and c2 be two constants used to maintain stability and avoid division by zero. This loss function measures the structural similarity between the enhanced image and the high-quality image. By minimizing the loss function, the enhanced image can be made as close as possible to the high-quality image in terms of brightness, contrast, and structure.

[0047] Figure 4 The images shown are the results of a specific embodiment based on the method of the present invention. The three images on the left are low-contrast thermal infrared images, while the three images on the right are high-quality images enhanced by the method of the present invention, which have high overall contrast and no distortion of local details and textures.

Claims

1. A thermal infrared image enhancement method based on global and local feature fusion modulation, characterized in that, A low-quality thermal infrared image is input into a thermal infrared image enhancement model. Through the collaborative efforts of a feature extraction module and an image enhancement backbone module, the model fully utilizes global and local information to enhance the image, outputting a thermal infrared image with good visual effects. The method for obtaining the thermal infrared image enhancement model is as follows: S1: Dataset Construction: Construct a pseudo-infrared image dataset with high and low quality pairings; S2: Construction of thermal infrared image enhancement model; S21: Construct a feature extraction module, which processes low-quality pseudo-infrared images to generate feature tensors; S22: Construct an image enhancement backbone module. The image enhancement backbone module uses the global and local feature modulation parameters obtained by the feature tensor transformation to perform global and local fusion modulation on the feature map of the low-quality pseudo-infrared image, which is then used as the model output. S23: Using high-quality pseudo-infrared images as ground truth, supervise the model output of step S22 based on the structural similarity loss function, train the feature extraction module and the image enhancement backbone module until convergence, and obtain the thermal infrared image enhancement model. The feature extraction module includes a global feature extraction module and a local feature extraction module; The global feature extraction module sequentially includes three convolutional layers with 32 channels each and a global average pooling layer. Each convolutional layer is followed by a ReLU activation function. The first convolutional layer has a kernel size of 7×7 and a stride of 2. The second and third convolutional layers both have a kernel size of 3×3 and a stride of 2. The output of the global feature extraction module is a global feature vector, which includes the global brightness and contrast information of the input image. The local feature extraction module comprises three convolutional layers with 32 channels each, each followed by a ReLU activation function. The three convolutional layers are cascaded together, with the output of each layer serving as the input to the next. The first convolutional layer has a 7×7 kernel size and a stride of 1, while the second and third convolutional layers have 3×3 kernel sizes and a stride of 1. The output of the local feature extraction module is a three-dimensional local feature matrix, which includes rich local structure and contrast information. The image enhancement backbone module comprises three convolutional layers with 64 channels each, two ReLU activation functions, and six fully connected layers. In each convolutional layer, all kernels are 1×1 with a stride of 1, used to independently process each pixel in the image. The fully connected layers transform the global feature vector and the local feature 3D matrix to obtain global and local feature modulation parameters. After each convolutional layer, a global feature modulation layer and a local feature modulation layer are provided. Features are fed into the local feature modulation layer after passing through the global feature modulation layer. The global and local feature modulation layers use scaling and offset operations to modulate the intermediate features output by the convolutional layers. The global feature modulation layer is modeled as follows: ,in yes Eigenvalues ​​at coordinate positions and These are global feature modulation parameters obtained from the transformation of the global feature vector, namely the global scaling factor and the offset factor; the local feature modulation layer is modeled as... ,in yes The result after processing by the global feature modulation layer and These are the local feature modulation parameters obtained from the transformation of the local feature three-dimensional matrix, representing respectively... Scaling factor and offset factor of eigenvalues ​​at coordinate positions.

2. The thermal infrared image enhancement method based on global and local feature fusion modulation according to claim 1, characterized in that, In step S1, the dataset construction is based on the existing visible light high-low quality paired dataset. A large number of high-low quality paired pseudo-infrared images are simulated and generated to obtain a high-low quality paired pseudo-infrared image dataset. The specific method is as follows: First, the images in the visible light high-low quality paired dataset are converted into grayscale images. Then, the pixel values ​​of the grayscale images are further adjusted to make them closer to the real thermal infrared images, thus obtaining preliminary pseudo-infrared images. Next, for a set of preliminary high-low quality paired pseudo-infrared images, the contrast of the high-quality images is further enhanced using various methods, and the image with the highest contrast is selected as the high-quality pseudo-infrared image. At the same time, the low-quality images are degraded using a random contrast compression method to obtain images with different degrees of degradation as low-quality pseudo-infrared images. Each low-quality pseudo-infrared image is then paired with a corresponding high-quality pseudo-infrared image to obtain multiple pairs of high-low quality paired pseudo-infrared images.

3. The thermal infrared image enhancement method based on global and local feature fusion modulation according to claim 1, characterized in that, The feature extraction module extracts global and local information from different scene images by generating feature tensors, which are then used by the subsequent image enhancement backbone module.

4. The thermal infrared image enhancement method based on global and local feature fusion modulation according to claim 1, characterized in that, The structural similarity loss function is defined as follows: ,in For the enhanced image result, For high-quality images as supervision, yes The average value, yes variance yes The average value, yes variance yes and covariance, and These are two constants used to maintain stability and avoid division by zero.

5. The thermal infrared image enhancement method based on global and local feature fusion modulation according to claim 1, characterized in that, When the thermal infrared image enhancement model is deployed into a thermal infrared imaging system, the thermal infrared imaging system can output thermal infrared images with good visual effects in real time.

Images