Infrared image enhancement method based on deep learning

Abstract

The application provides an infrared image enhancement method based on deep learning, and steps are as follows: collecting clean infrared images and adding different types and intensities of noises to the clean infrared images to construct a data set; using a data enhancement technology to expand the data set; designing a new DnCNN model: introducing a Cbc convolution module in the first layer of the DnCNN network and introducing an ETBF feature fusion module in the last layer; using a training set to train the new DnCNN model, using a verification set to adjust the hyperparameters of the new DnCNN model and monitoring the performance of the new DnCNN model; using the trained DnCNN model to carry out denoising processing on infrared images in a test set to remove noises in the infrared images; using a DPW-SwinIR model to carry out deep convolution and point-by-point convolution on the infrared images subjected to the denoising processing, and obtaining enhanced high-resolution infrared images. The application can accurately remove noises in infrared images and restore detailed information covered by the noises, and has a wide application prospect in the field of infrared image processing.

Description

Technical Field

[0001] This invention relates to the technical field of infrared image enhancement, and more particularly to an infrared image enhancement method based on deep learning. Background Technology

[0002] Infrared imaging is a type of image that uses infrared radiation to obtain information about the thermal distribution of a target object. Compared to visible light images, infrared images can penetrate atmospheric interference such as fog and smoke, giving them the advantage of long-distance imaging in dark or harsh environments. Therefore, infrared imaging has wide applications in military, security, and medical fields.

[0003] The importance of infrared image enhancement lies in improving the quality and detail of infrared images, making them clearer, easier to analyze, and understand. In target detection and recognition, infrared images are commonly used in military reconnaissance and security surveillance. Enhancement of infrared images can effectively improve target discrimination and recognition accuracy, helping military and security personnel quickly locate and position targets, thus enhancing combat and security effectiveness. In medical diagnosis and treatment, infrared images are increasingly widely used in the medical field. Enhancement of infrared images can improve doctors' ability to detect and diagnose diseases or lesions. In industry and non-destructive testing, infrared images are widely used. Enhancement of infrared images can help engineers and technicians detect material defects, thermal failures, and other problems, improving product quality and safety.

[0004] The invention patent with publication number CN115409726A provides an infrared image enhancement method based on multi-scale information exchange. First, convolutional learning is performed on the original-size infrared image to obtain a coarse feature map. Then, the coarse feature map is upsampled and downsampled to extract features at different scales. Next, a random selection and fusion module is introduced to exchange feature information at different scales, achieving cross-scale feature learning. Finally, feature information from multiple scales is combined to preserve the spatial details of the original image. This invention can better supplement the feature learning process at the original scale and improve the image's detail representation. However, this method uses a random selection and fusion module to exchange feature information at different scales, which may lead to some differences in the processing results each time, thus affecting the final image enhancement effect. Chinese patent application CN115082318A discloses a method for super-resolution reconstruction of infrared images of electrical equipment. It improves the ESRGAN model by introducing an improved batch normalization module and a new feature extraction submodule. These two modules are combined to construct a feature extraction network within a generative adversarial network (GAN). The improved batch normalization module is also incorporated into the ESRGAN GAN to enhance its discriminative power. This invention, using this improved method, enables the generative network to reconstruct higher-quality, higher-resolution infrared images of electrical equipment from low-resolution images. However, the ESRGAN model itself has a long processing time, and the introduction of the improved batch normalization module and the new feature extraction submodule may further increase the processing time, limiting its efficiency and speed in practical applications.

[0005] In recent years, the field of infrared image enhancement has been continuously developing. However, it still faces several challenges: 1. Current infrared image enhancement methods have limited effectiveness in processing low signal-to-noise ratio images. Due to the inherent characteristics of infrared images, noise has a significant impact on image quality. 2. Infrared images are greatly affected by ambient lighting conditions, such as sunlight and background radiation. Current enhancement methods often cannot effectively handle images under different lighting conditions, leading to unstable or over-enhanced results. 3. Infrared images often contain complex scenes and objects, such as strong textures and complex backgrounds. Existing methods often perform poorly in handling these complex scenes, making it difficult to accurately recover image details and structure. 4. The evaluation of infrared image enhancement often relies on subjective perception, lacking objective evaluation metrics to measure the quality of enhancement. This makes comparison and performance evaluation between different methods difficult. 5. Because infrared image enhancement algorithms typically require complex calculations and operations, practical applications often require long processing times and substantial computational resources. This limits the application of this technology in certain real-time scenarios. Summary of the Invention

[0006] To address the shortcomings of existing infrared image enhancement methods in handling complex scenes, accurately restoring image details, and the computationally complex nature of these methods, this invention proposes a deep learning-based infrared image enhancement method. Breaking away from traditional methods, this approach utilizes a specific structured DnCNN (Denoising Convolutional Neural Network) model, incorporates innovative concepts such as Coordinate-based convolution (Cbc) and ETBF (Edge Texture Features and Base Features Fusion) modules, and integrates a unique DPW (Depthwise Convolution and Pointwise Convolution Fusion) module. This enables accurate detection of noise in infrared images and restoration of details obscured by noise, bringing a significant breakthrough to the field of infrared image processing.

[0007] To achieve the above objectives, the technical solution of the present invention is implemented as follows: an infrared image enhancement method based on deep learning, comprising the following steps:

[0008] Step 1: Collect clean infrared images and add different types and intensities of noise to the clean infrared images to form image pairs, and construct a dataset based on the image pairs;

[0009] Step 2: Use data augmentation techniques to augment the dataset, and then divide the augmented dataset into training, validation, and test sets;

[0010] Step 3: Design a novel DnCNN model: Introduce a Cbc convolution module in the first layer of the DnCNN network, introduce an ETBF feature fusion module in the last layer, and use convolution and batch normalization in the middle layers. The ETBF feature fusion module fuses the generated basic features of the model with the edge texture features of the input infrared image.

[0011] Step 4: Train the novel DnCNN model using the training set to learn the feature representation of surface information in infrared images; adjust the hyperparameters of the novel DnCNN model and monitor its performance using the validation set.

[0012] Step 5: Use the trained DnCNN model to denoise the infrared images in the test set to remove noise from the infrared images;

[0013] Step 6: Use the DPW-SwinIR model to perform depthwise convolution and pointwise convolution on the denoised infrared image to obtain an enhanced high-resolution infrared image.

[0014] Preferably, each image pair includes a noisy image and a corresponding clean infrared image; the noise is Gaussian noise, salt-and-pepper noise, or Poisson noise, and the intensity of the noise is achieved by controlling the variance, density, or intensity parameters of the noise; the data augmentation technique is one or more of rotation, scaling, translation, and flipping.

[0015] Preferably, the Cbc convolution module processes the input infrared image as follows: First, it performs a channel-by-channel 1×1 convolution on the input infrared image to obtain an original feature map T. Then, it downsamples the original feature map T by a factor to generate multiple sub-feature maps K. A convolution operation with a stride of 1 is applied to the sub-feature maps K to traverse and capture more detailed information. The detailed information of each sub-feature map K is concatenated to form a stacked structure to obtain a sub-feature map H. Finally, a pruning method is used to reduce the number of channels in the sub-feature map H.

[0016] Preferably, the pruning method sorts the channels based on their weights, then selects and retains the channels with higher weight importance, while pruning the channels with lower weight importance.

[0017] Preferably, the pruning method is as follows:

[0018] By setting a threshold q, channels with lower importance metric values ​​im are pruned, while channels with higher importance metric values ​​im are retained.

[0019] Calculate the channel pruning ratio: prune_ratio = desired_channels / total_channels; where prune_ratio represents the channel pruning ratio, desired_channels represents the number of channels to be retained, and total_channels represents the total number of channels in the sub-feature map H.

[0020] Calculate the number of channels retained after pruning:

[0021] pruned_channels=total_channels-prune_ratio×total_channels;

[0022] The weight parameters after pruning are:

[0023] adjusted_weights=reshape(weights,[pruned_channels,-1]);

[0024] Here, weights represents the weight parameters before pruning, [pruned_channels,-1] represents the new dimension, and reshape is the deformation operation function.

[0025] Preferably, the method for downsampling the original feature map T to generate multiple sub-feature maps K is as follows: proportionally cutting the positions on the original feature map T that satisfy the conditions (i+c) and (j+d) can be divided by the proportion, thereby generating multiple sub-feature maps K; where (i, j) are the pixel coordinates of the first channel in the original feature map T, and c and d are the intervals between horizontal and vertical pixels.

[0026] The importance metric for a channel is im = sum(abs(W)); where W represents the weight parameter vector corresponding to the channel, abs represents the absolute value function, and sum represents the summation function.

[0027] Through multiple rounds of iterative training, the weight parameters are continuously adjusted; the weight parameters of the channels before pruning are learned during the training process using the gradient descent algorithm.

[0028] The method for fine-tuning the new DnCNN model after pruning is as follows: use the adjusted weight parameters adjusted_weights as the initial weight parameters to further train the entire new DnCNN model;

[0029] The novel DnCNN model has a total of 20 layers, with ReLU as the activation function. Layers 2 to 19 are convolutional with batch normalization. Each layer has 64 convolutional kernels and a kernel size of 3×3.

[0030] Preferably, the edge texture features of the clean infrared image are extracted by an edge texture extraction algorithm, which includes an infrared image edge extraction algorithm based on omnidirectional gradient and a non-maximum suppression algorithm. The infrared image edge extraction algorithm based on omnidirectional gradient extracts the input infrared image to generate an edge texture image, and the edge texture image is processed by the non-maximum suppression algorithm to generate an edge texture feature map.

[0031] Preferably, the method for generating edge texture feature maps by processing the edge texture image using a non-maximum suppression algorithm is as follows:

[0032] The gradient magnitude and gradient direction in the edge texture image are calculated using the Canny edge detection algorithm.

[0033] Scanning the edge texture image: For each pixel, perform the following operations: a. Determine if the current pixel is a local maximum: Find two adjacent pixels of the current pixel along the gradient direction; b. Calculate the gradient magnitude ratio: Calculate the gradient magnitude ratio of the current pixel based on the two adjacent pixels along the gradient direction; c. Non-maximum suppression: If the gradient magnitude ratio is less than the gradient magnitude of the current pixel, set the gradient magnitude of the current pixel to 0; d. Thresholding: Compare the gradient magnitudes of all pixels after non-maximum suppression with a given threshold. When the gradient magnitude of a pixel is greater than the given threshold, it is considered an edge; e. Post-processing: Perform morphological operations or other post-processing operations on the obtained binarized edge map to obtain the edge texture feature map Y.

[0034] Preferably, the implementation steps of the ETBF feature fusion module are as follows:

[0035] 1) The base feature map X and edge texture feature map Y generated by the previous layer of the ETBF feature fusion module of the novel DnCNN model are used as input;

[0036] 2) The basic feature map X and edge texture feature map Y generated by the previous layer of the ETBF feature fusion module of the novel DnCNN model are mapped into a common low-dimensional vector representation by a self-attention mechanism. The mapped vectors are feature map x and feature map y, respectively.

[0037] 3) Use the Gaussian kernel function as the interpolation method to interpolate feature map x and feature map y respectively, to obtain feature map A and feature map B respectively;

[0038] 4) Divide feature map A and feature map B into multiple sub-regions, and then perform feature fusion on each sub-region layer by layer;

[0039] 5) Perform spatial pyramid pooling on the feature matrix after layer-by-layer fusion to obtain the final fused feature vector Z.

[0040] Preferably, the method for obtaining the enhanced high-resolution infrared image using the DPW-SwinIR model is as follows:

[0041] (1) Perform depthwise convolution on the denoised infrared image, using a convolution kernel of size D. K ×D K ×M, where D KM represents the width and height of the convolution kernel, and M represents the depth of the convolution kernel. After the denoising process, each channel of the infrared image is generated by depth convolution to produce an output channel. The output channels are added together to obtain the feature map. (2) Perform pointwise convolution: use N convolution kernels of size 1×1×Q to fuse the feature map obtained by depth convolution. The convolution kernel of size 1×1×Q is used to weight the feature maps of different channels to generate a new feature map. The number of channels of the output feature map is Q.

[0042] Compared with existing technologies, the beneficial effects of this invention are as follows: First, a novel DnCNN (Denoising Convolutional Neural Network) model is designed. The DnCNN model has a specific structure and an appropriate number of layers to effectively process infrared images. Second, a Cbc (Coordinate-based convolution) module and an ETBF (Edge Texture Features and Base Features Fusion) feature fusion module are innovatively introduced. This not only better captures spatial information and structural features in the image but also matches them with the details that need enhancement, improving the accuracy of noise detection in the image. Finally, combining this with a SwinIR (Image Restoration Using Swin Transformer) model incorporating a DPW (Depthwise Convolution and Pointwise Convolution Fusion) module helps to recover details in the image that were originally obscured by noise. This invention is targeted, innovative, and efficient in infrared image enhancement, accurately removing noise and restoring details obscured by noise, and has broad application prospects in the field of infrared image processing. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart of the present invention.

[0045] Figure 2 This is a structural diagram of the Cbc convolution module of the present invention.

[0046] Figure 3 This is a schematic diagram of the convolution factor of the present invention.

[0047] Figure 4 This is a flowchart of the ETBF feature fusion module of the present invention. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] like Figure 1 As shown, a deep learning-based infrared image enhancement method analyzes the noise distribution, edge, and structural features of infrared images to construct a deep learning model suitable for infrared image enhancement. Inspired by advances in visible light research, this invention uses deep learning to enhance infrared images to obtain high-quality infrared images. This invention performs the first step of denoising infrared images by independently constructing a novel DnCNN model. Firstly, the structure of the DnCNN model is not constant; different denoising requirements can be met by adjusting the number of network layers, activation functions, and connection methods. This flexibility is a significant advantage of deep learning models, enabling the DnCNN model to adapt to diverse tasks and data. The novel DnCNN model consists of 20 layers, all using ReLU activation functions, which are computationally simple, saving computation time and overcoming some gradient vanishing issues. This invention innovatively proposes a Cbc convolution module, which can better capture global and local feature patterns, making even small, noisy objects in the image readily visible. An ETBF feature fusion module is introduced before the last layer to improve the model's multi-scale feature extraction capability. Finally, the DPW-SwinIR model is used to perform super-resolution reconstruction of the denoised infrared image, restoring high-resolution details.

[0050] This invention constructs a novel DnCNN model suitable for infrared image enhancement. The structure and number of layers of the DnCNN model are determined by considering both the model's denoising effect and processing time. A novel Cbc convolution module and an ETBF feature fusion module are introduced, which help to better capture spatial information and structural features in the image and match them with the details that need enhancement, thereby improving the accuracy of noise detection. An improved infrared image edge extraction method based on omnidirectional gradients is proposed, optimizing the edge response map so that the extracted edge texture feature image is more suitable for the ETBF feature fusion module. The DnCNN model is combined with the DPW-SwinIR image super-resolution reconstruction model. Due to the removal of noise, the DPW-SwinIR image super-resolution reconstruction model can better focus on the super-resolution reconstruction task and effectively recover the details in the image that were originally obscured by noise. The specific steps of the implementation method of this invention are as follows:

[0051] Step 1: Collect clean infrared images, which can be obtained from open datasets or relevant research institutions. Image pair construction typically involves overlaying clean infrared images with noisy images of appropriate types and intensities to form the input image and the corresponding target image. Each image pair includes one noisy image and its corresponding clean image. The type of noise added can be determined based on the specific scenario and requirements, including Gaussian noise, salt-and-pepper noise, Poisson noise, etc. The intensity can be controlled by adjusting the variance, density, or intensity parameters of the noise. By constructing a dataset containing clean infrared images and noisy image pairs, the generalization ability, robustness, and anti-interference ability of the DnCNN model can be improved, and a reliable benchmark can be provided for model training and performance evaluation.

[0052] Step 2: When constructing the dataset, consider the following key points: Dataset Size: The size of the dataset should be determined based on the actual situation, generally the larger the better, to provide more comprehensive and sufficient samples. Dataset Partitioning: Divide the dataset into training, validation, and test sets. The training set is used for model training, the validation set is used for tuning hyperparameters and monitoring model performance, and the test set is used to evaluate the model's generalization ability. Data Augmentation: Data augmentation techniques can be used to expand the dataset, such as rotation, scaling, translation, and flipping, to increase data diversity and robustness.

[0053] Step 3: Design a novel DnCNN model specifically for infrared image enhancement: Introduce a Cbc convolution module in the first layer of the DnCNN network, introduce an ETBF feature fusion module in the last layer, and use convolution and batch normalization in the middle layers. The ETBF feature fusion module fuses the model's basic features and edge and texture features.

[0054] The infrared image depth denoising network proposed in this invention is an optimization and improvement based on DnCNN. The network has a total of 20 layers, and all activation functions are ReLU. The first layer of the new DnCNN model introduces a Cbc convolution module, and the last layer introduces an ETBF feature fusion module.

[0055] The convolutional operations in the first layer of the DnCNN network play a role in image denoising tasks, including preprocessing, feature extraction, noise removal, and the introduction of nonlinear transformations. This provides the foundation for subsequent denoising layers and helps the DnCNN network better understand and process the input noisy image. Given the importance of this part, this invention first chooses to make improvements in this part.

[0056] To better capture global and local feature patterns and make noisy small objects in images impossible to hide, the first layer of the novel DnCNN model uses a Cbc convolution mode to replace the traditional stride convolution and pooling operations. The processing of the Cbc convolution module is as follows: Figure 2 As shown.

[0057] The specific steps of the Cbc convolution module are as follows: First, the input infrared image is convolved 1×1 channel by channel to obtain the original feature map T. Then, the obtained feature map T is downsampled. By proportionally cutting the positions on the feature map that satisfy the conditions (i+c) and (j+d) are divisible by the proportion, multiple sub-feature maps K are generated. Here, (i, j) are the pixel coordinates of the first channel in the feature map, S represents the vertical and horizontal dimensions of the feature map, and b is the number of channels. Here, the size of the feature map is S×S×b; c and d are the intervals between horizontal and vertical pixels. For ease of understanding, it is assumed that c = d = 1. Therefore, after downsampling, the size of the feature map is (S / 2)×(S / 2)×b, and K = 4 sub-feature maps are obtained, as shown below. Figure 2 As shown, each sub-feature map K is obtained by downsampling the original feature map T by a factor of 1. This reduces the size of the feature map while retaining important information. Next, a convolution operation with a stride of 1 is applied to the downsampled sub-feature map K. This means that as each convolution kernel slides across the sub-feature map K, it traverses and captures more detailed information, thus providing a more effective feature representation. The collected sub-feature maps K are concatenated to form a stacked structure, namely the sub-feature map H. The stacked sub-feature maps can provide richer information representation, multi-scale information, improve model robustness, and bring advantages such as parameter sharing and computational efficiency, which help improve the model's performance and expressive power. Finally, a pruning method is used to further reduce the number of channels in the stacked structure. Based on the importance weight of each channel, it is determined which channels to retain and which unimportant channels to prune. The resulting features of the pruning method mainly include three aspects:

[0058] A streamlined model, better generalization ability, and more concise feature representation. Pruning removes unnecessary feature channels, thereby reducing model size and improving computational efficiency. Simultaneously, pruning reduces model complexity and enhances generalization ability. The final model retains only feature channels that contribute significantly to the classification results, thus improving the model's understanding and representation of input data. This invention sorts channels based on their weights, then selects and retains channels with higher weight importance while pruning channels with lower weight importance. The channel pruning calculation method used in this invention can be summarized as follows:

[0059] The importance of a channel is measured as shown in Equation (1).

[0060] im = sum(abs(W)) (1)

[0061] Here, `im` represents the channel importance metric, and `W` represents the weight parameter vector corresponding to that channel. Through multiple rounds of iterative training, the weight parameters are continuously adjusted, enabling the network to better fit the training data and exhibit good generalization ability. The trained weight parameters can be seen as key parameters for the network to extract and represent features during the learning process, determining the network's output. In pruning methods, the importance metric based on the weight parameters can be used to evaluate the importance of channels, thereby performing pruning operations to remove unimportant channels, thus reducing model size and improving computational efficiency. `abs` represents the absolute value function, and `sum` represents the summation function.

[0062] The importance metric 'im' is used to determine the number of channels to retain. By setting a threshold 'q', channels with lower importance metrics are pruned, while channels with higher importance metrics are retained.

[0063] The pruning ratio of the channel is calculated as shown in formula (2).

[0064] prune_ratio=desired_channels / total_channels (2)

[0065] Where prune_ratio represents the channel pruning ratio, desired_channels represents the number of channels to be retained, and total_channels represents the total number of channels in the sub-feature map H.

[0066] The number of channels after pruning is shown in formula (3).

[0067] pruned_channels=total_channels-prune_ratio×total_channels (3)

[0068] Here, pruned_channels represents the number of channels retained after pruning.

[0069] The dimensional adjustment of the weight parameters after pruning is shown in formula (4).

[0070] adjusted_weights=reshape(weights,[pruned_channels,-1]) (4)

[0071] Here, `adjusted_weights` represents the adjusted weight parameters, and `weights` represents the weight parameters before pruning, which are elements of the weight parameter vector W. In a neural network, each convolutional layer consists of multiple channels, and each channel contains a set of weight parameters used to perform convolution operations on the input data. The weight parameters of the channels before pruning are learned during training using the gradient descent algorithm. `[pruned_channels,-1]` represents the new dimension. `reshape` is the deformation function, which adjusts the original weight parameter tensor to a new shape through the `reshape` operation.

[0072] The pruned model needs fine-tuning to restore performance. In this case, the adjusted weights can be used as the initial weights, and then the entire model can be trained further. This fine-tuning stage aims to adapt the model to the new parameters and maintain high performance.

[0073] This invention, by employing the convolution method of the Cbc convolution module, can better handle tasks involving low-resolution images and small objects, which helps to improve the robustness and discriminative ability of the novel DnCNN model to features.

[0074] The novel DnCNN model uses convolutional layers (2-19) with batch normalization (Bn) and ReLU. Batch normalization accelerates network convergence and improves generalization. Each layer has 64 convolutional kernels of 3×3 size, adding non-linear transformations while maintaining the receptive field. This allows for better capture of image details, thus improving model performance. The novel DnCNN model has 20 layers, increasing its complexity and enabling better data fitting, further enhancing performance. The 20-layer depth also allows the model to learn more detailed and abstract features, further improving performance. All activation functions use ReLU, which is simple, fast, and stable. Using ReLU in DnCNN makes training easier and avoids gradient vanishing. Furthermore, ReLU increases the model's non-linearity, allowing for better data fitting and improved performance.

[0075] The final layer of the DnCNN model also plays a crucial role, restoring and adjusting the model's output to refine and optimize it for specific task requirements. To compensate for potential information loss during the preceding 19 convolutional layers, a feature fusion module is introduced before layer 20, providing a more complete and robust feature representation. This module primarily fuses the basic features extracted by the DnCNN model with features extracted by a novel edge texture extraction algorithm.

[0076] First, we introduce the edge texture feature extraction algorithm. This invention employs an improved infrared image edge extraction algorithm based on omnidirectional gradient, which can effectively extract edge and texture information in images and has the following characteristics:

[0077] 1. Flexible convolution kernel radius: The algorithm uses a flexible convolution kernel radius for calculations. By adjusting the kernel size, the degree of edge and texture extraction can be controlled, making the edge and texture feature extraction algorithm suitable for image analysis needs in different scenarios. (See diagram for convolution kernel radius illustration.) Figure 3 As shown.

[0078] 2. Omnidirectional Gradient Synthesis: This method synthesizes gradient values ​​from multiple directions to obtain an omnidirectional gradient. Compared to traditional edge extraction algorithms, the use of omnidirectional gradients can more accurately capture the direction and changes of edges.

[0079] 3. Eliminates the need for manual thresholding: Traditional edge extraction algorithms typically require manually setting thresholds to determine the extraction result, while the omnidirectional gradient-based infrared image edge extraction algorithm overcomes this drawback. It adaptively determines the edge extraction threshold during the calculation process, thus extracting edge and texture information more intelligently.

[0080] 4. Combining the omnidirectional gradient infrared image edge extraction algorithm with the non-maximum suppression (NMS) algorithm effectively eliminates redundant edges in the edge response map and refines the edges to a single pixel width, thus representing the edge information in the image more accurately. Furthermore, the high efficiency of the NMS algorithm improves computational efficiency, making the edge extraction process faster and more efficient. Finally, the output edge texture feature map can be used as input to a subsequent feature fusion module for further analysis and processing of the edge features of the infrared image.

[0081] The infrared image edge extraction algorithm based on omnidirectional gradient is combined with the non-maximum suppression (NMS) algorithm. The specific implementation steps are as follows:

[0082] 1. Input the edge texture image generated by the infrared image edge extraction algorithm based on omnidirectional gradient.

[0083] 2. Calculate the gradient magnitude and gradient direction in the edge texture image: The Canny edge detection algorithm has the advantages of high accuracy, real-time performance and low false alarm rate. Therefore, the gradient magnitude and gradient direction of the image are calculated based on the Canny edge detection algorithm. The gradient magnitude represents the edge intensity of the pixel, and the gradient direction represents the direction of the edge.

[0084] 3. Scanning the Edge Texture Image: For each pixel, perform the following operations: a. Determine if the current pixel is a local maximum: Find two adjacent pixels of the current pixel along the gradient direction. b. Calculate the gradient magnitude ratio (Ratio): Calculate the gradient magnitude ratio of the current pixel based on the two adjacent pixels along the gradient direction. c. Non-maximum suppression: If the gradient magnitude ratio (Ratio) is less than the gradient magnitude of the current pixel, set the gradient magnitude of the current pixel to 0 to suppress non-maximum values. d. Thresholding: Compare the gradient magnitudes of all pixels after non-maximum suppression with a given threshold. Only when the gradient magnitude of a pixel is greater than the given threshold is it considered an edge. A suitable threshold is automatically determined using the maximum inter-class variance. e. Post-processing: Perform morphological operations or other post-processing operations on the obtained binarized edge map to eliminate noise and broken edges, thereby obtaining the edge texture feature map Y.

[0085] like Figure 4 As shown, the implementation steps of the ETBF feature fusion module can be summarized as follows:

[0086] 1. Take the basic feature map X (generated by the nineteenth layer of the DnCNN model) and the edge texture feature map Y as input.

[0087] 2. The two feature maps are mapped separately using a self-attention mechanism, transforming them into a common low-dimensional vector representation. Let the mapped vectors be feature map x and feature map y, respectively. By introducing a self-attention mechanism, the mapping process becomes more flexible and can better capture important information from the feature maps.

[0088] 3. To better recover details in the image, a Gaussian kernel function is used as the interpolation method to interpolate feature map x and feature map y, respectively, resulting in feature map A and feature map B. Using the Gaussian kernel function as a non-linear interpolation method allows for better modeling of complex relationships by introducing non-linear weights. Specifically, the Gaussian kernel function assigns different weights to different pixels based on the distance between the target location and its neighboring pixels. Pixels that are closer to the target location receive higher weights, while pixels that are farther away receive lower weights.

[0089] 4. Divide feature map A and feature map B into multiple sub-regions, and then perform feature fusion on each sub-region layer by layer. Layer-by-layer fusion can better capture the local structural information of different feature maps and avoid the fuzziness and confusion that may be caused by global feature fusion. The division can be regular, such as dividing into 8 sub-regions on average. The division involves separating and processing different feature maps separately, and then fusing the processed results.

[0090] 5. Perform spatial pyramid pooling on the fused feature matrix to obtain the final fused feature vector Z. Spatial pyramid pooling can handle cases where the input feature map has different sizes. It divides the input feature map into multiple sub-regions of different sizes, performs pooling operations on each sub-region, and finally concatenates all pooling results. This captures the spatial information of features at different scales, giving the model better scale invariance and context awareness. Here, the splitting involves dividing the input feature map into multiple sub-regions of different sizes, performing pooling operations on each sub-region, and finally concatenating all pooling results.

[0091] The ETBF feature fusion module of the present invention captures the correlation and contextual information between feature maps while retaining detailed information through steps such as mapping, interpolation and fusion, thereby achieving effective feature fusion. The whole process can extract more expressive and richer features.

[0092] Step 4: Train the novel DnCNN model from Step 3 using the training set to learn the feature representation of surface information in infrared images.

[0093] Provide a specific implementation plan:

[0094] 1. Select 1000 labeled infrared images from the dataset as the training set.

[0095] 2. Perform preprocessing on each image, including normalizing pixel values ​​to the [0,1] range and adding Gaussian noise.

[0096] 3. Choice of Loss Function: Mean Squared Error (MSE) is chosen as the loss function. The mean squared error between the DnCNN model output and the true label is calculated. MSE is a common loss function that can help the model learn pixel-level feature representations better.

[0097] 4. Selection of Optimization Method: Select the Adam optimizer, and set the learning rate to 0.001, beta1 to 0.9, and beta2 to 0.999. Adam is a commonly used adaptive optimization algorithm that can automatically adjust the learning rate based on the gradient.

[0098] 5. Use of validation and test sets: Divide the training set into 800 images as the training set and 200 images as the validation set.

[0099] 6. Training Process: Input the training set into the DnCNN model for training, setting the batch size to 64 and iteratively training for 100 epochs. During training, adjust hyperparameters and network structure based on the performance on the validation set, such as adding / removing convolutional layers or changing the kernel size. Update model parameters using the Adam optimizer, calculate the MSE loss for each batch, and record the loss values ​​and validation set performance metrics (such as PSNR, SSIM, etc.) during training.

[0100] Step 5: Use the trained DnCNN model to denoise the infrared image to remove noise from the image.

[0101] After the DnCNN model is trained, the test set is an independent set of samples retained during the training process, used to evaluate the performance of the trained model on unseen data.

[0102] Step 6: Use the DPW-SwinIR model to further optimize the denoised image by performing depthwise convolution and pointwise convolution to obtain an enhanced high-resolution infrared image.

[0103] The SwinIR model is a neural network model for image super-resolution reconstruction. It allows for adjustment of the super-resolution factor and achieves good performance in computer vision tasks. The shallow feature extraction part of the SwinIR model uses only a single 3×3 convolutional layer. The convolution process involves... F ×DF The input of ×M is converted to D F ×D F The output is ×N, and only the number of channels changes after convolution, where M represents the number of input channels and N represents the number of output channels.

[0104] For infrared images with complex content, to improve the model's enhancement effect on infrared images, DPW convolution operation is used in the shallow feature extraction part. Assume the size of the input feature map is D. F ×D F ×M, where D F The width and height of the feature map are represented by M, and the number of channels is represented by M. The processing method of the DPW-SwinIR model is as follows: (1) Perform depthwise convolution on the denoised image. The depthwise convolution uses a kernel size of D. K ×D K ×M, where D K M represents the width and height of the convolution kernel, and M represents the depth of the convolution kernel (the same as the number of channels in the denoised image). The number of convolution kernels is 1. The operation of depthwise convolution is to convolve each channel with the convolution kernel. Each channel will generate an output channel. Then these output channels are added together to obtain the final output feature map. (2) Then pointwise convolution is performed. N 1×1×Q convolution kernels are used to fuse the feature map obtained by depthwise convolution. These 1×1×Q convolution kernels are used to weight the feature maps of different channels to generate a new feature map. The final output feature map has Q channels. One of the functions of pointwise convolution is to fuse the feature map obtained by depthwise convolution with 1×1×Q convolution kernels and adjust the number of channels of the feature map according to the requirements. Through the weighting operation of pointwise convolution, the feature maps of different channels can be combined and adjusted to generate a new feature map with Q channels. Therefore, the number of channels can be dynamically adjusted through pointwise convolution.

[0105] Depthwise convolution helps capture features from each channel, while pointwise convolution helps fuse information from different channels to produce higher-level feature representations. Combining depthwise and pointwise convolution can improve computational efficiency, increase feature diversity, and provide flexibility in network design while preserving spatial information.

[0106] This invention inputs a denoised image into the DPW-SwinIR model for optimization. Since noise has been removed from the image, the DPW-SwinIR model can better focus on the super-resolution reconstruction task, utilizing details previously obscured by noise to generate higher-quality, clearer, higher-resolution images. This invention is the first to combine a denoising model with an image super-resolution reconstruction model. By combining the denoising capabilities of the DnCNN model with the super-resolution reconstruction capabilities of the DPW-SwinIR model, better image enhancement results can be achieved.

[0107] This invention employs a 20-layer DnCNN model structure, capable of capturing multi-scale information, enhancing feature representation capabilities, and achieving excellent denoising performance without overfitting. In the first layer, this invention abandons the traditional convolutional model and introduces a Cbc convolution mode. By downsampling and segmenting the feature map into multiple sub-feature maps, the feature map size is reduced while retaining important information, making it difficult for noise and small objects in the image to survive. For the stacked sub-feature maps, this invention also introduces the idea of ​​channel pruning, determining which channels to retain and prune based on their weights, thereby reducing the number of model parameters and computational complexity. Furthermore, this invention introduces an ETBF feature fusion module before the last layer of the DnCNN model. Combined with edge and texture extraction algorithms, it can completely extract the edge and texture features of infrared images, compensating for the loss of edge, texture, and structural information that may have been caused by the preceding 19 convolutional layers, providing a more complete and robust feature representation. Finally, the DPW-SwinIR model is used to further optimize the denoised image. Since noise has been removed, the DPW-SwinIR model can better recover high-resolution details, marking the first application of super-resolution reconstruction methods in infrared image enhancement. This innovative infrared image enhancement method based on deep learning has enormous potential and broad application prospects in the field of image processing.

[0108] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A deep learning-based infrared image enhancement method, characterized in that, The steps are as follows: Step 1: Collect clean infrared images and add different types and intensities of noise to the clean infrared images to form image pairs, and construct a dataset based on the image pairs; Step 2: Use data augmentation techniques to augment the dataset, and then divide the augmented dataset into training, validation, and test sets; Step 3: Design a novel DnCNN model: Introduce a Cbc convolution module in the first layer of the DnCNN network, introduce an ETBF feature fusion module in the last layer, and use convolution and batch normalization in the middle layers. The ETBF feature fusion module fuses the generated basic features of the model with the edge texture features of the input infrared image. Step 4: Train the newly designed DnCNN model using the training set to learn the feature representation of surface information in infrared images; We used the validation set to tune the hyperparameters of the novel DnCNN model and monitor its performance. Step 5: Use the trained DnCNN model to denoise the infrared images in the test set to remove noise from the infrared images; Step 6: Use the DPW-SwinIR model to perform depthwise convolution and pointwise convolution on the denoised infrared image to obtain an enhanced high-resolution infrared image.

2. The infrared image enhancement method based on deep learning according to claim 1, characterized in that, Each image pair includes a noisy image and a corresponding clean infrared image; the noise is Gaussian noise, salt-and-pepper noise, or Poisson noise, and the intensity of the noise is achieved by controlling the variance, density, or intensity parameters of the noise; the data augmentation technique is one or more of rotation, scaling, translation, and flipping.

3. The infrared image enhancement method based on deep learning according to claim 1 or 2, characterized in that, The Cbc convolution module processes the input infrared image as follows: First, it performs a channel-by-channel 1×1 convolution on the input infrared image to obtain the original feature map T. Then, it downsamples the original feature map T by a factor to generate multiple sub-feature maps K. A convolution operation with a stride of 1 is applied to the sub-feature maps K to traverse and capture more detailed information. The detailed information of each sub-feature map K is concatenated to form a stacked structure to obtain a sub-feature map H. Finally, a pruning method is used to reduce the number of channels in the sub-feature map H.

4. The infrared image enhancement method based on deep learning according to claim 3, characterized in that, The pruning method sorts the channels based on their weights, then selects and retains the channels with higher weight importance, while pruning the channels with lower weight importance.

5. The infrared image enhancement method based on deep learning according to claim 4, characterized in that, The pruning method is as follows: By setting a threshold q, channels with lower importance metric values ​​im are pruned, while channels with higher importance metric values ​​im are retained. Calculate the channel pruning ratio: prune_ratio = desired_channels / total_channels; where prune_ratio represents the channel pruning ratio, desired_channels represents the number of channels to be retained, and total_channels represents the total number of channels in the sub-feature map H. Calculate the number of channels retained after pruning: pruned_channels=total_channels-prune_ratio×total_channels; The weight parameters after pruning are: adjusted_weights=reshape(weights,[pruned_channels,-1]); Here, weights represents the weight parameters before pruning, [pruned_channels,-1] represents the new dimension, and reshape is the deformation operation function.

6. The infrared image enhancement method based on deep learning according to claim 5, characterized in that, The method for generating multiple sub-feature maps K by downsampling the original feature map T by a factor is as follows: the original feature map T is cut into positions that satisfy the conditions (i+c) and (j+d) can be divided by the proportion, thereby generating multiple sub-feature maps K; where (i,j) are the pixel coordinates of the first channel in the original feature map T, and c and d are the intervals between horizontal and vertical pixels. The importance metric for a channel is im = sum(abs(W)); where W represents the weight parameter vector corresponding to the channel, abs represents the absolute value function, and sum represents the summation function. Through multiple rounds of iterative training, the weight parameters are continuously adjusted; the weight parameters of the channels before pruning are learned during the training process using the gradient descent algorithm. The method for fine-tuning the new DnCNN model after pruning is as follows: use the adjusted weight parameters adjusted_weights as the initial weight parameters to further train the entire new DnCNN model; The novel DnCNN model has a total of 20 layers, with ReLU as the activation function. Layers 2 to 19 are convolutional with batch normalization. Each layer has 64 convolutional kernels and a kernel size of 3×3.

7. The infrared image enhancement method based on deep learning according to any one of claims 1, 2, 4-6, characterized in that, The edge texture features of the clean infrared image are extracted by an edge texture extraction algorithm, which includes an infrared image edge extraction algorithm based on omnidirectional gradient and a non-maximum suppression algorithm. The infrared image edge extraction algorithm based on omnidirectional gradient extracts the input infrared image to generate an edge texture image, and the edge texture image is processed by the non-maximum suppression algorithm to generate an edge texture feature map.

8. The infrared image enhancement method based on deep learning according to claim 7, characterized in that, The method for generating edge texture feature maps by processing the edge texture image using a non-maximum suppression algorithm is as follows: The gradient magnitude and gradient direction in the edge texture image are calculated using the Canny edge detection algorithm. Scanning edge texture image: For each pixel, perform the following operations: a. Determine if the current pixel is a local maximum: Find the two adjacent pixels of the current pixel in the gradient direction; b. Calculate the gradient magnitude ratio: Based on two adjacent pixels along the gradient direction, calculate the gradient magnitude ratio of the current pixel; c. Non-maximum suppression: If the gradient magnitude ratio is less than the gradient magnitude of the current pixel, then the gradient magnitude of the current pixel is set to 0; d. Thresholding: The gradient magnitudes of all pixels after non-maximum suppression are compared with a given threshold. When the gradient magnitude of a pixel is greater than the given threshold, it is considered an edge; e. Post-processing: Morphological operations or other post-processing operations are performed on the obtained binarized edge map to obtain the edge texture feature map Y.

9. The infrared image enhancement method based on deep learning according to claim 8, characterized in that, The implementation steps of the ETBF feature fusion module are as follows: 1) The base feature map X and edge texture feature map Y generated by the previous layer of the ETBF feature fusion module of the novel DnCNN model are used as input; 2) The basic feature map X and edge texture feature map Y generated by the previous layer of the ETBF feature fusion module of the novel DnCNN model are mapped into a common low-dimensional vector representation by a self-attention mechanism. The mapped vectors are feature map x and feature map y, respectively. 3) Use the Gaussian kernel function as the interpolation method to interpolate feature map x and feature map y respectively, to obtain feature map A and feature map B respectively; 4) Divide feature map A and feature map B into multiple sub-regions, and then perform feature fusion on each sub-region layer by layer; 5) Perform spatial pyramid pooling on the feature matrix after layer-by-layer fusion to obtain the final fused feature vector Z.

10. The infrared image enhancement method based on deep learning according to claim 8 or 9, characterized in that, The method for obtaining the enhanced high-resolution infrared image using the DPW-SwinIR model is as follows: (1) Perform depthwise convolution on the denoised infrared image, using a kernel size of D. K ×D K ×M, where D K M represents the width and height of the convolution kernel, and M represents the depth of the convolution kernel. After the denoising process, each channel of the infrared image is generated by depth convolution to produce an output channel. The output channels are added together to obtain the feature map. (2) Perform pointwise convolution: use N convolution kernels of size 1×1×Q to fuse the feature map obtained by depth convolution. The convolution kernel of size 1×1×Q is used to weight the feature maps of different channels to generate a new feature map. The number of channels of the output feature map is Q.

Images