An infrared image enhancement method based on a deep fully convolutional neural network
An infrared image enhancement network was constructed by using a deep fully convolutional neural network. By utilizing convolutional blocks and residual structures, the problem of image quality degradation caused by noise amplification in existing infrared image enhancement methods was solved, and high-quality infrared image enhancement and recognition performance were improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SOUTHWEAT UNIV OF SCI & TECH
- Filing Date
- 2022-04-15
- Publication Date
- 2026-05-08
AI Technical Summary
Existing infrared image enhancement methods tend to amplify background noise while enhancing the target, and the edges are easily blurred. They also make it difficult to fully utilize the temperature information of infrared images, especially in low-resolution situations where recognition tasks are challenging.
A deep fully convolutional neural network-based approach is adopted to construct an infrared image enhancement network. By utilizing convolutional blocks and residual structures, combined with data preprocessing and data augmentation, an infrared image enhancement model is trained to achieve high-quality infrared image enhancement.
It effectively suppresses background noise, preserves image edge details, is suitable for infrared thermal imaging cameras of different resolutions, improves image quality and recognition performance, reduces computing resource consumption, and is suitable for low-computing environments.
Smart Images
Figure CN116977188B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of infrared image technology, specifically to an infrared image enhancement method based on a deep fully convolutional neural network. Background Technology
[0002] With the advent of automated mobile equipment for goods, identification in nighttime and unstable extreme conditions has become increasingly common. This need has led to the use of multimodal sensors that can complement each other. Thermal cameras offer a rich source of temperature information and are less affected by changing lighting or background clutter. However, existing thermal cameras have relatively lower resolution than RGB cameras, making it difficult to fully utilize this information in identification tasks. Therefore, enhancing infrared images to obtain high-quality infrared images is highly beneficial for infrared thermal imaging camera-based identification tasks.
[0003] Traditional image enhancement methods, such as histogram equalization (HE), often amplify background noise while enhancing the target image, resulting in poor performance. An improved algorithm, contrast-limited adaptive histogram equalization (CLAHE), can suppress background noise, but edges are easily blurred. To mitigate this issue, this invention, based on extensive analysis of existing methods, implements a low-resolution thermal image enhancement method based on deep convolutional neural networks. Summary of the Invention
[0004] In view of the above-mentioned technical shortcomings, the present invention provides an infrared image enhancement method based on a deep fully convolutional neural network, comprising the following steps:
[0005] S1, acquire and create infrared high and low gain image pairs to obtain a rich training sample set and test set;
[0006] S2, depending on the different acquisition methods, performs corresponding data preprocessing and data augmentation on the training sample set obtained in S1;
[0007] S3 constructs an infrared image enhancement network by introducing convolutional blocks and residual structures;
[0008] S4. The training sample set obtained in S2 is fed into the network designed in S3 for training to obtain the infrared image enhancement model.
[0009] S5. Input the infrared test image obtained in S1 into the infrared image enhancement model trained in S4 to obtain the enhanced infrared image.
[0010] As a preferred embodiment, the two methods for obtaining infrared high- and low-gain image pairs described in S1 are as follows:
[0011] Method 1: Synthesize grayscale infrared images. Use visible light images from public datasets and perform grayscale processing on these images to obtain high-gain grayscale images. The grayscale processing method is as follows: use a random contrast function with a contrast factor range of [0.5, 0.51] to perform low-contrast processing on the high-gain grayscale images to obtain the corresponding low-gain grayscale images;
[0012] Method 2: Using pseudo-color infrared images, a commercial infrared camera with high and low gain modes is used to collect high and low gain infrared image pairs for different scenes. The obtained infrared image pairs form a dataset.
[0013] Preferably, the data preprocessing operation described in S2 is as follows:
[0014] Step 1: Use the sliding window cropping algorithm to crop the original image pair, fix and reduce the size of the infrared image, and make the image length and width equal.
[0015] Step two involves performing data enhancement on the segmented image using methods such as image flipping, image contrast transformation, and image scaling.
[0016] Preferably, the infrared image enhancement network constructed in S3 includes the following:
[0017] The image feature extraction section utilizes eight convolutional layers to extract infrared image features, resulting in eight feature maps. Each convolutional layer is followed by the LeakyReLU activation function, the mathematical expression of which is as follows:
[0018]
[0019] Next, residual structures are used to fuse feature maps, so that the feature maps contain rich details and semantic information; two skip connections are used in the eight convolutional layers to fully fuse the features of each layer;
[0020] Finally, the output is obtained using a single convolutional layer, which is followed by the LeakyReLU activation function.
[0021] As a preferred embodiment, the structure of the convolutional block in S3 is as follows:
[0022] For the first to fourth convolutional layers, the kernel size is set to 7×7, the stride is set to 1, and the padding is set to 3.
[0023] For the fifth convolutional layer, the kernel size is set to 5×5, the stride is set to 1, and the padding is set to 2.
[0024] For the sixth to ninth convolutional layers, the kernel size is set to 3×3, the stride is set to 1, and the padding is set to 3.
[0025] For all convolutional blocks, padding is filled with 0.
[0026] As a preferred embodiment, the mathematical expression for the residual structure described in S3 is as follows:
[0027]
[0028] in, x and y These represent the input and output, respectively. f ( x ) represents the feature extraction layer; a residual structure is introduced to fuse the features extracted by the first convolutional layer and the fourth convolutional layer of the infrared image enhancement model, as well as the features extracted by the fifth convolutional layer and the eighth convolutional layer.
[0029] Preferably, the hyperparameter settings for the training process described in S4 are as follows:
[0030] (1) The initial learning rate lr is 1e-3. Using cosine annealing decay, the learning rate decreases from 1e-3 to 1e-9. The cosine annealing formula is:
[0031]
[0032] in, ηt The current learning rate, ηmax To achieve the maximum learning rate, ηmin To minimize the learning rate, Tcur This represents the current iteration number. Tmax This represents the maximum number of iterations.
[0033] (2) Optimization algorithm selection Adam Gradient descent was used, with the momentum parameter set to 0.9, and mean squared loss was employed as the loss function. The optimization process employed... Adam Gradient descent and general backpropagation methods are used to measure PSNR (Peak Signal-to-Noise Ratio) during training to check the progress of model training. Adam The algorithm strategy can be expressed as:
[0034]
[0035] Where, m t and v tThese are the first-order momentum and second-order momentum terms, respectively. β 1, β 2 represents the power value, with values of 0.9 and 0.999. , These are their respective correction values. W t express t At time t, i.e., the parameters of the model in the t-th iteration, g t =△J(W t )express t The gradient magnitude of the cost function with respect to W in the next iteration; ε It is a very small number, usually 1. e -8 is used to avoid the denominator being 0;
[0036] (3) The batch size is set to 64, the total iteration period EPOCH is set to 500, the total number of training samples is set to S, and the maximum number of iterations N is set as follows:
[0037]
[0038] S41, Input an infrared image of a set size. After normalization, the resolution of the image obtained by each convolutional layer remains unchanged.
[0039] S42, then three convolutional modules are used to further extract deep features, and the output at this time is fused with the features extracted by the first convolutional layer, so as to effectively preserve the texture information in the shallow features and avoid degradation problems.
[0040] S43, then the fifth convolution is performed to continue extracting features;
[0041] S44, finally three convolution operations are performed, and the output at this time is fused with the features extracted by the fifth convolutional layer using the residual structure to obtain a feature map rich in semantic and detailed information, which is the final output;
[0042] S45, calculate the loss value of the enhanced infrared image relative to the reference infrared image. The loss function uses the mean square error function, the mathematical expression of which is:
[0043]
[0044] Where m represents the batch size, and W and H represent the image length and width. f ij This represents the pixel value at coordinate point (i, j) in the reference infrared image. f ij ' represents the pixel value of the predicted augmented infrared image coordinate point (i, j);
[0045] S46. Based on the calculated loss value, the gradient is calculated in reverse to update the parameters of the infrared image enhancement network.
[0046] S47. Repeat the above steps until the training count reaches the set maximum number of iterations N, then stop training; save the trained infrared enhancement model.
[0047] The beneficial effects of this invention are:
[0048] 1. Image processing methods based on deep neural networks currently rely heavily on resource availability. Two methods are used to collect and create training and testing data to enable deep neural network models that depend on the number of samples to fully realize their performance.
[0049] 2. The fully convolutional neural network based on the deep convolutional neural network design allows for unlimited input image size and can be applied to infrared thermal imaging cameras of different resolutions;
[0050] 3. The network is designed with a relatively lightweight and simple structure to produce high-quality output. It is easy to implement, saves training and inference time, and can be used in practical applications in low-computation environments.
[0051] 4. By introducing a residual structure, the model can better fit the mapping relationship between high-quality and low-quality infrared images, thus obtaining high-quality infrared images. Attached Figure Description
[0052] Figure 1 Here is a flowchart provided by the present invention;
[0053] Figure 2 Here is a diagram of the infrared enhancement network structure provided by this invention;
[0054] Figure 3 Here is a schematic diagram of the residual structure provided by this invention;
[0055] Figure 4 Here is a schematic diagram of the convolutional block provided by the present invention; Detailed Implementation
[0056] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to Figures 1-4.
[0057] Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also...
[0058] The details in this specification can also be implemented or applied through other specific embodiments, and can be based on different viewpoints.
[0059] Various modifications or changes may be made to the application without departing from the spirit of the invention. Based on the embodiments of the present invention, those in the art...
[0060] All other embodiments obtained by a person skilled in the art without inventive effort are protected by this invention.
[0061] scope.
[0062] like Figure 1 As shown, an infrared image enhancement method based on a deep fully convolutional neural network includes the following steps:
[0063] S1, acquire and create infrared high and low gain image pairs to obtain a rich training sample set and test set;
[0064] In practical implementation, two methods are used to obtain infrared high and low gain image pairs:
[0065] Method 1: Synthesize grayscale infrared images. Use visible light images from public datasets and perform grayscale processing on these images to obtain high-gain grayscale images. The grayscale processing method is as follows: use a random contrast function with a contrast factor range of [0.5, 0.51] to perform low-contrast processing on the high-gain grayscale images to obtain the corresponding low-gain grayscale images;
[0066] Method 2: Using pseudo-color infrared images, a commercial infrared camera with high and low gain modes is used to collect high and low gain infrared image pairs for different scenes. The obtained infrared image pairs form a dataset.
[0067] The image pairs obtained by the two methods are merged, and the dataset is divided into a training sample set and a test sample set in an 8:2 ratio.
[0068] S2, depending on the different acquisition methods, performs corresponding data preprocessing and data augmentation on the training sample set obtained in S1;
[0069] Step one: Use a sliding window cropping algorithm to crop the original image pair, fixing and reducing the size of the infrared image.
[0070] Make the image have equal length and width;
[0071] For the training sample set, to reduce computer resource consumption, a sliding window algorithm is used to crop the infrared images in the training sample set to a window size of 128×128, with a stride of 40×40, thus fixing the infrared image size to 128×128. This not only reduces the computational load of subsequent models, thereby reducing the time used in the model training phase, but also enriches the infrared image training samples. For the test sample set, the infrared images in the training sample set are cropped to a window size of 256×256.
[0072] Step two involves performing data enhancement on the segmented image using methods such as image flipping, image contrast transformation, and image scaling.
[0073] S3 constructs an infrared image enhancement network by introducing convolutional blocks and residual structures;
[0074] The infrared image enhancement network constructed in S3 includes the following:
[0075] like Figure 2 As shown, the image feature extraction part utilizes eight convolutional layers to extract infrared image features, resulting in eight feature maps. The shallow features extracted by the first convolutional layer contain more details, while the feature map extracted by the eighth convolutional layer contains more deep semantic information. Each convolutional layer is followed by the LeakyReLU activation function, with the negative_slope parameter set to 0.01. The mathematical expression of the LeakyReLU activation function is as follows:
[0076]
[0077] The LeakyReLU activation function is simple and computationally fast. Then, using a residual structure, the output of the first convolutional layer is fused into the output of the fourth convolutional layer, and the output of the fifth convolutional layer is fused into the eighth layer, resulting in feature maps containing rich detailed and semantic information.
[0078] like Figure 3 As shown, the mathematical expression for the residual structure in S3 is:
[0079]
[0080] in, and These represent the input and output, respectively. This represents the feature extraction layer. As the network depth increases, the objective function becomes increasingly prone to getting trapped in local optima. Simultaneously, the vanishing gradient problem worsens with increasing layer count, preventing the network parameters of the output layer from being effectively learned. Residual structures are a highly effective method to alleviate the vanishing gradient problem, significantly increasing the depth of networks that can be effectively trained. Residual units can be implemented using skip connections, where the unit's input is directly added to its output before activation. Therefore, skip connections are used to connect the features extracted by the first convolutional layer of the infrared image enhancement model to the fourth layer, and the features extracted by the fifth convolutional layer to the eighth layer, to avoid gradient vanishing and gradient explosion. Furthermore, using residual structures reduces training complexity and facilitates backpropagation.
[0081] like Figure 4As shown, the structure of the convolutional block in S3 is as follows:
[0082] For the first to fourth convolutional layers, the kernel size is set to 7×7, the stride is set to 1, and the padding is set to 3.
[0083] For the fifth convolutional layer, the kernel size is set to 5×5, the stride is set to 1, and the padding is set to 2.
[0084] For the sixth to ninth convolutional layers, the kernel size is set to 3×3, the stride is set to 1, and the padding is set to 3.
[0085] For all convolutional blocks, padding is filled with 0;
[0086] In the first to fourth layers, the kernel size is set to 7×7 in order to expand the receptive field and obtain more information;
[0087] A stride of 1 is set to achieve higher extraction accuracy; simultaneously, padding is set to 3 to ensure that the resolution of the image obtained by each convolutional layer remains constant. From the fifth to the ninth layer, the kernel size is progressively reduced to minimize the number of parameters while fully fusing low-level features. Zeros are padded before convolutions to ensure that the size of all feature maps is the same as the size of the input map.
[0088] S4. Use the training sample set obtained in S2 to supervise the training of the infrared image enhancement network and obtain the infrared image enhancement model.
[0089] The supervised training process in S4 is as follows:
[0090] S41, Input an infrared image of a set size. After normalization, the resolution of the image obtained by each convolutional layer remains unchanged.
[0091] S42, then three convolutional modules are used to further extract deep features, and the output at this time is fused with the features extracted by the first convolutional layer, so as to effectively preserve the texture information in the shallow features and avoid degradation problems.
[0092] S43, then the fifth convolution is performed to continue extracting features;
[0093] S44, finally three convolution operations are performed, and the output at this time is fused with the features extracted by the fifth convolutional layer using the residual structure to obtain a feature map rich in semantic and detailed information, which is the final output;
[0094] S45, calculate the loss value of the enhanced infrared image relative to the reference infrared image. The loss function uses the mean square error function, the mathematical expression of which is:
[0095]
[0096] Where m represents the batch size, and W and H represent the image length and width. f ij This represents the pixel value at coordinate point (i, j) in the reference infrared image. f ij ' represents the pixel value of the predicted augmented infrared image coordinate point (i, j);
[0097] S46. Based on the calculated loss value, the gradient is calculated in reverse to update the parameters of the infrared image enhancement network.
[0098] S47, Repeat the above steps until the training count reaches the set maximum number of iterations N, then stop training; save the trained infrared enhancement model;
[0099] The hyperparameter settings for the training process in S4 are as follows:
[0100] (1) The initial learning rate lr is 1e-3. Using cosine annealing decay, the learning rate decreases from 1e-3 to 1e-9. The cosine annealing formula is:
[0101]
[0102] in, ηt The learning rate for the current period. ηmax To maximize the learning rate, ηmin To minimize the learning rate, Tcur This represents the current iteration number. Tmax This represents the maximum number of iterations.
[0103] (2) The Adam gradient descent algorithm was selected for optimization, with the momentum parameter set to 0.9 and the mean squared loss function used. The optimization process employed the Adam gradient descent method and a general backpropagation scheme. During training, the PSNR (Peak Signal-to-Noise Ratio) was measured simultaneously to check the progress of model training. The Adam algorithm strategy can be expressed as:
[0104]
[0105] Where, m t and v t These are the first-order momentum and second-order momentum terms, respectively. β 1, β 2 represents the power value, with values of 0.9 and 0.999. , These are their respective correction values. W t express t At time t, i.e., the parameters of the model in the t-th iteration, g t =△J(W t )express t The gradient magnitude of the cost function with respect to W in the next iteration; ε It is a very small number, usually 1. e -8 is used to avoid the denominator being 0;
[0106] (3) The batch size is set to 64, the total iteration period EPOCH is set to 500, the total number of training samples is set to S, and the maximum number of iterations N is set as follows:
[0107]
[0108] The learning rate is a crucial hyperparameter in deep learning, controlling how quickly the weights of a neural network are adjusted based on the loss gradient. A smaller learning rate results in slower gradient descent and a longer convergence time. Too small a learning rate leads to very slow gradient descent, while too large a learning rate may cause the learning rate to exceed the optimal value. Choosing appropriate hyperparameters during training can accelerate the training process, improve training speed, and enhance model performance.
[0109] S5 inputs the low-quality infrared test image into the infrared image enhancement model trained in S4 to obtain the enhanced infrared image.
[0110] External image;
[0111] The convolutional and fully connected layers in the designed model have parameters; the training process involves adjusting these parameters to achieve the desired model performance.
[0112] The output image closely resembles a real, high-quality image. During training, the predicted high-quality image and the real image are compared...
[0113] High-quality image distance difference (mean squared error function) can reveal the quality of the model in each iteration; the smaller the loss, the better the prediction.
[0114] The closer to a high-quality image, the better. Backpropagation uses the loss error chain rule to calculate the gradient, and then updates the model parameters accordingly.
[0115] The number refers to the kernel parameters of the convolutional layer. One iteration includes one forward propagation and one backward propagation.
[0116] The test image is compared with the enhanced image, and the Peak Signal-to-Noise Ratio (PSNR), a commonly used metric in image enhancement, is used as the performance evaluation standard. The formula for calculating PSNR is as follows:
[0117]
[0118]
[0119] Where I is a clean image with a resolution of m×n, K is a noisy image of the same size, MSE is the image error, and MAX2I is the maximum possible pixel value of the image.
[0120] Under the same conditions, the comparison results with other image enhancement methods are shown in Table 1:
[0121] Table 1 Comparison with other methods
[0122]
[0123] As can be seen, compared with methods such as histogram equalization (HE), TEN, TIECNN, and IE-GAN, the method of this patent has higher performance indicators and better results, proving the effectiveness of the method of this patent.
[0124] The above description is merely a preferred embodiment of the present invention, and it should be understood that the present invention is not limited to the forms disclosed herein.
[0125] This should not be construed as excluding other embodiments, but can be used in various other combinations, modifications, and environments, and is capable of being described herein.
[0126] Within the scope of this concept, modifications can be made using the teachings above or techniques or knowledge in related fields. Modifications made by those skilled in the art are also permitted.
[0127] Any modifications or variations that do not depart from the spirit and scope of this invention shall be within the protection scope of the appended claims.
Claims
1. An infrared image enhancement method based on a deep fully convolutional neural network, characterized in that, Includes the following steps: S1, acquire and create infrared high and low gain image pairs to obtain a rich training sample set and test set; S2, depending on the different acquisition methods, performs corresponding data preprocessing and data augmentation on the training sample set obtained in S1; S3 constructs an infrared image enhancement network by introducing convolutional blocks and residual structures; S4. The training sample set obtained in S2 is fed into the network designed in S3 for training to obtain the infrared image enhancement model. S5. Input the infrared test image obtained in S1 into the infrared image enhancement model trained in S4 to obtain the enhanced infrared image. The infrared image enhancement network constructed in S3 includes the following: The image feature extraction part uses eight convolutional layers to extract infrared image features, resulting in eight feature maps. Each convolutional layer is followed by the LeakyReLU activation function, the mathematical expression of which is shown in Equation 1. Next, residual structures are used to fuse feature maps, so that the feature maps contain rich details and semantic information; two skip connections are used in the eight convolutional layers to fully fuse the features of each layer; Finally, the output is obtained using a single convolutional layer, which is followed by the LeakyReLU activation function. The convolutional block structure described in S3 is as follows: For the first to fourth convolutional layers, the kernel size is set to 7×7, the stride is set to 1, and the padding is set to 3. For the fifth convolutional layer, the kernel size is set to 5×5, the stride is set to 1, and the padding is set to 2. For the sixth to ninth convolutional layers, the kernel size is set to 3×3, the stride is set to 1, and the padding is set to 3. For all convolutional blocks, padding is filled with 0; The mathematical expression for the residual structure described in S3 is shown in Equation 2: in, x and y These represent the input and output, respectively. f ( x () represents the feature extraction layer; Skip connections are used to connect the features extracted from the first convolutional layer of the infrared image enhancement model to the fourth layer, and the features extracted from the fifth convolutional layer to the eighth layer, in order to avoid gradient vanishing and gradient explosion. At the same time, using residual structures can reduce training complexity and facilitate backpropagation.
2. The infrared image enhancement method based on a deep fully convolutional neural network according to claim 1, characterized in that, The method for acquiring and creating infrared high- and low-gain image pairs described in S1 is as follows: Method 1: Synthesize grayscale infrared images by using visible light images from public datasets and converting these images to grayscale to obtain high-gain grayscale images; Method 2: Using pseudo-color infrared images, a commercial infrared camera with high and low gain modes is used to collect infrared image pairs for different scenes, and all infrared image pairs are used to form a dataset.
3. The infrared image enhancement method based on a deep fully convolutional neural network according to claim 1, characterized in that, The data preprocessing and augmentation operations described in S2 are as follows: S31, The original image pairs are cropped using a sliding window cropping algorithm to make the image length and width equal and to expand the training sample pairs; S32 uses image flipping, image contrast transformation, and image scaling to augment the training samples.
4. The infrared image enhancement method based on a deep fully convolutional neural network according to claim 1, characterized in that, The hyperparameter settings for the training process described in S4 are as follows: (1) Initial learning rate lr 1 e -3, using cosine annealing decay, the learning rate starts from 1. e -3 dropped to 1 e -9; The cosine annealing formula is shown in Formula 3: in, ηt The current learning rate, ηmax To maximize the learning rate, ηmin To minimize the learning rate, Tcur This represents the current iteration number. Tmax This represents the maximum number of iterations. (2) The Adam gradient descent algorithm was selected for optimization, with the momentum parameter set to 0.9 and the mean square loss function used. The optimization process employed the Adam gradient descent method and a general backpropagation scheme. During training, the PSNR (Peak Signal-to-Noise Ratio) was measured simultaneously to check the progress of model training. The Adam algorithm strategy can be expressed as Equation 4: in, m t and v t These are the first-order momentum and second-order momentum terms, respectively. β 1, β 2 represents the power value, with values of 0.9 and 0.
999. , Each is a correction value; W t express t Time, i.e., the first t Parameters of the iterative model g t =△J(W t () represents the magnitude of the gradient of the cost function with respect to W in t iterations; ε It is a very small number, usually 1. e -8 is used to avoid the denominator being 0; (3) The batch size is set to 64, the total iteration period EPOCH is set to 500, the total number of training samples is set to S, and the maximum number of iterations N is as shown in Formula 5: 。 5. The infrared image enhancement method based on a deep fully convolutional neural network according to claim 1, characterized in that, The specific process of S4 is as follows: S41, Input an infrared image of a set size. After normalization, the resolution of the image obtained by each convolutional layer remains unchanged. S42, then three convolutional modules are used to further extract deep features, and the output at this time is fused with the features extracted by the first convolutional layer, so as to effectively preserve the texture information in the shallow features and avoid degradation problems. S43, then the fifth convolution is performed to continue extracting features; S44, finally three convolution operations are performed, and the output at this time is fused with the features extracted by the fifth convolutional layer using the residual structure to obtain a feature map rich in semantic and detailed information, which is the final output; S45, calculate the loss value of the enhanced infrared image relative to the reference infrared image. The loss function uses the mean square error function, the mathematical expression of which is shown in Formula 6: Where m represents the batch size, and W and H represent the image length and width. f ij This represents the pixel value at coordinate point (i, j) in the reference infrared image. f ij 'Represents the pixel value of the predicted augmented infrared image coordinate point (i, j); S46. Based on the calculated loss value, the gradient is calculated in reverse to update the parameters of the infrared image enhancement network. S47. Repeat the above steps until the training count reaches the set maximum number of iterations N, then stop training; save the trained infrared enhancement model.