A method for infrared image denoising and small target detection in severe weather conditions

Abstract

The present invention discloses a method for infrared image denoising and small target detection in severe weather conditions. First, a three-category infrared image small target dataset is constructed, the dataset is randomly divided into three equal parts, and three types of noise, namely Gaussian noise, Poisson noise, and salt and pepper noise, are added; then, a network architecture for infrared image denoising and small target detection in severe weather conditions is constructed, the overall framework includes three parts: a filter denoising module, a full-resolution denoising module, and a YOLOv5 detector. The loss supervision function is used in the detector to improve the detection performance of small targets; finally, based on the noisy three-category infrared image small target dataset, the proposed network architecture is trained to obtain a model for denoising and recognition of infrared images. The denoising module of the present invention can remove noise and extract the area where the key features are located in the image, and then the key area is passed into the detector to obtain the positioning information and category information of the target, which can better improve the performance of small target detection.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and provides a method for denoising infrared images and detecting small targets under adverse weather conditions. Background Art

[0002] Infrared image object detection plays a vital role in transportation, healthcare, and the military. This class of algorithms targets infrared images. Object detection and recognition in infrared images has the ability to detect targets in dim environments, improving target recognition performance at night and in adverse weather conditions. Unlike natural images, targets in infrared images are typically smaller and more blurred, making them more challenging to identify against cluttered backgrounds. While deep learning-based object detection methods have achieved promising results on traditional datasets, localizing targets in low-quality images captured under adverse weather conditions remains challenging. Existing methods either struggle to balance image enhancement and object detection or often overlook potential information that could aid detection. This paper proposes that images captured under adverse weather conditions can be decomposed into a clean image and its corresponding weather-specific information. The degradation of image quality under adverse weather conditions is primarily caused by the interaction between weather-specific information and objects, resulting in poor detection performance. Infrared image denoising still faces many challenges. Traditional methods require a clear image prior for training, resulting in numerous parameters and complex computation. Single-image dehazing based on convolutional neural networks aims to estimate a clear image given a blurred input and is not well-suited to varying blur. Image priors are difficult to apply to different types of real-world images, as real haze is more complex than modeled. Existing deep learning models use fixed learning weights, requiring the network to use the same weights to remove varying fog and haze concentrations. They also typically use a large number of filters to map blurry inputs to deblurred results, increasing computational overhead. Existing algorithms primarily focus on locating target locations, with relatively simple classification schemes. Therefore, further research is needed on infrared image denoising and small target detection in inclement weather.

[0003] Inclement weather conditions include rain, fog, and snow. In natural images, rain, snow, and fog appear directly in their original form. Infrared images are acquired using infrared thermal imaging technology, which uses infrared thermal imaging instruments to convert the temperature distribution of a target surface into an image visible to the human eye, displaying the target surface temperature distribution in varying grayscale colors. Therefore, in infrared images, rain, snow, and fog cannot be recorded in their original form; instead, they are recorded as grayscale features within the background image. By observing the characteristic distribution of rain, snow, and fog in infrared images, we consider them captured by infrared thermal imaging instruments as different types of noise, which facilitates the study of small target detection and classification tasks in infrared images used under inclement weather conditions.

[0004] Rain, fog, and snow can appear differently when captured by infrared cameras, but all can result in some blurry or peculiar features in the image. These features appear differently in the infrared spectrum than in the visible spectrum. Fog can have an appearance similar to Gaussian noise, appearing as a uniform, fuzzy feature in infrared images, similar to the soft blurring effect Gaussian noise can create. Rain can also have an appearance similar to Poisson noise, with the distribution of raindrops exhibiting a Poisson-like pattern, resulting in regularly spaced dark areas in the image. Snow can also have an appearance similar to salt and pepper noise, a random noise pattern that introduces black and white pixels into the image, similar to the visual effect of snowflakes. Summary of the Invention

[0005] In view of the deficiencies of the prior art, the present invention provides an infrared image denoising and small target detection method for use in adverse weather conditions.

[0006] This method combines infrared image denoising and small target detection and classification modules end-to-end, using joint training for small target detection and recognition in infrared images captured by infrared cameras in adverse weather. The method consists of three components: a filter denoising module, a full-resolution denoising module, and a detector module. The input image is first fed into the filter denoising module and the full-resolution denoising module, respectively. The features from the two different denoising modules are then fused to obtain the final denoised feature information. The denoised feature information is then passed to the detector to obtain the final detection and classification results. The classic one-stage detector YOLOv5 is selected as the baseline detector to improve its performance under adverse conditions. This method proposes an end-to-end mixed data training scheme with a detection loss. This scheme combines the advantages of filters and deep neural networks in denoising tasks to enhance images for target detection in a weakly supervised manner. The dataset used in this method is a self-built three-category infrared image dataset (GAS-Noise), a dataset of small targets in three-category noisy infrared images. This dataset includes various scenes, including blurred shapes, overlapping objects, and various obstacles. GAS-Noise includes three types of targets. The dataset for aerial targets comes from the IST-A dataset, which was published by Xu Hai in the paper "Multi-Scale Multi-Level Residual Feature Fusion for Real-Time Infrared Small Target Detection". The datasets for vehicle targets and ship targets are from the PaddlePaddle AI website. We first unified the image format of the three collected datasets and the XML document tag format of the target box, and then used labelImg software to check and verify the annotation box of each image in the dataset, re-labeled the missed and mislabeled targets, and saved the new annotation document. Finally, all images were divided into three equal parts and Gaussian noise, Poisson noise, and salt and pepper noise were randomly added, with a noise intensity of 10dB.

[0007] The technical solution adopted by the present invention to solve the technical problem includes the following steps:

[0008] Step 1. Based on the task objectives of infrared image denoising and detection and recognition, a three-category infrared image small target dataset is constructed. The dataset is randomly divided into three equal parts and three types of noise are added: Gaussian noise, Poisson noise, and salt and pepper noise. The noise intensity is 10dB.

[0009] Step 2. Build a network architecture for infrared image denoising and small target detection in adverse weather conditions. The overall framework consists of three parts: a filter denoising module, a full-resolution denoising module, and a YOLOv5 detector. The detector uses a loss supervision function to improve the detection performance of small targets.

[0010] Step 3. Based on the noisy three-category infrared image small target dataset, the proposed network architecture is trained to obtain a model for denoising and recognition of infrared images.

[0011] Step 4. Test the trained infrared image denoising and small target detection model with the test set to verify the model's accuracy (AP) and average precision (mAP) indicators.

[0012] Furthermore, the specific method of step 1 is as follows:

[0013] According to the task objectives of infrared image denoising and small target detection, infrared images suitable for small target detection and recognition tasks were collected and sorted under aerial background, ground background and ocean background respectively. The label box of each image in the dataset was checked using labelimg software. Missing and mislabeled targets were re-labeled and new annotation documents were saved. The formats of infrared images and annotation files were unified. The image format was JPG and the annotation file format was JSON. The image resolution was 384*288 and 640*480. Based on the above infrared images, all images were randomly divided into three equal parts and three types of noise were added: Gaussian noise, Poisson noise and salt and pepper noise. The noise intensity was 10dB. The three-category infrared image small target dataset with noise was obtained for the experiment of the present invention. Finally, the number of classification samples, the number of targets and the target size of the three-category infrared image small target dataset with noise were counted.

[0014] Furthermore, noises generated in bad weather such as rain, snow, and fog exist in the form of filters on infrared images. A filter denoising module is used to process local noise, which can also be called a local denoising module. The filter denoising step is to pass the image through a median filter, a wavelet denoising filter, and a sharpening filter in sequence to perform local denoising on the image. The median filter and wavelet denoising filter are used to process Gaussian noise, Poisson noise, and salt and pepper noise, and the sharpening filter is used to adjust the sharpness of the infrared image. In the case of discrete targets, the formula of the median filter is: median (x1, y1) = median (I (x1 + i, y1 + j) | i = 0, 1, ..., N-1, j = 0, 1, ...N-1)

[0015] Where median (x1, y1) is the pixel value of the image after median filtering, median represents the median of the pixel values ​​in the neighborhood, I(x1+i, y1+j) is the pixel value of the input image at position (x1+i, y1+j), and N is the size of the neighborhood. Wavelet denoising is an image denoising method based on wavelet transform. Wavelet transform decomposes the image into components of different frequencies, removes high-frequency noise through thresholding, and then performs an inverse transform. Using wavelet transform to decompose and reconstruct the image, the wavelet denoising method is written as:

[0016] Wavelet denoising (x1, y1) = W -1 (T(W(I(x1,y1))))

[0017] Among them, W is the wavelet transform, T is the threshold function, which is used to perform threshold processing on the wavelet coefficients, W -1 is the inverse wavelet transform, and I(x1, y1) is the output of the image features after the median filter, which is the pixel value at (x1, y1). After the median filter and wavelet denoising filter, the image is passed to the sharpening filter to adjust the contrast between the image background and the target. The sharpening filter is used to highlight image details. The formula is as follows:

[0018] F(x1, λ)=I(x1)+λ(I(x1)-Gau(I(x1)))

[0019] Here, I(x1) is the output of the wavelet denoising filter, representing the pixel value at coordinate x1, Gau() represents a Gaussian filter, and the parameter λ is a proportional factor. This sharpening operation is differentiable with respect to both x and λ, and the degree of sharpening can be adjusted by optimizing the parameter λ to achieve better object detection performance. The above three filters form a local denoising module. The median filter and wavelet denoising filter suppress rain, snow, and fog noise in inclement weather. The sharpening filter adjusts the sharpness of the infrared image, enhancing the key target area, which facilitates the detection and recognition of small targets in infrared images.

[0020] Furthermore, the full-resolution denoising module consists of 12 convolution-normalization-activation function blocks, a four-layer residual convolution block, and three skip links. The full-resolution denoising module is divided into three stages according to the data flow order: the downsampling stage, the transfer connection stage, and the upsampling stage. The downsampling stage is used for image feature extraction: the image input passes through two layers of convolution-normalization-activation function blocks and then undergoes convolution downsampling to obtain input feature A1. Input feature A1 passes through two layers of convolution-normalization-activation function blocks and then undergoes convolution downsampling to obtain input feature A2. Input feature A2 passes through two layers of convolution-normalization-activation function blocks and then undergoes convolution downsampling to obtain input feature A3. Then, the transfer connection stage begins: input feature A3 enters the four-layer convolution residual block in the middle layer to obtain the restored feature B3. The upsampling stage is used for denoising and feature restoration. To enhance feature sharing, features are concatenated: B3 and A3 are concatenated, interpolated and upsampled, and then pass through two layers of convolution-normalization-activation function blocks to obtain the restored feature B2. B2 and A2 are concatenated, interpolated and upsampled, and then pass through two layers of convolution-normalization-activation function blocks to obtain the restored feature B1. B1 and A1 are concatenated, interpolated and upsampled, and then pass through two layers of convolution-normalization-activation function blocks to obtain the final full-resolution denoised image. The formula for the full-resolution denoising module is as follows:

[0021] I coarse =FRM(I noise )

[0022] Among them I noise and I coarse Represents the original noisy infrared image and the image after the full-resolution denoising module. By using L2 loss, it is ensured that the full-resolution module can obtain the denoising result without pixel distortion:

[0023]

[0024] in, is the L2 loss, I bg Represents a noise-free infrared image.

[0025] Furthermore, the feature information obtained by the local denoising module and the global denoising module is fused to obtain the optimal denoised feature map, which is then passed to the detection network to facilitate the detection network's identification of the location and category information of small targets in infrared images. In the detection network, the YOLOv5 detector is used as the detection network. The YOLOv5 detector is divided into three parts: feature extraction module, feature fusion module, and detection module. The incoming feature information is sequentially passed through the feature extraction module, feature fusion module, and detection module. The feature extraction module randomly scales and crops the input image features and passes them to the backbone network containing the Focus structure and the CSP structure. The feature fusion module passes the image features to a combined top-down and bottom-up feature fusion network for fusion. The detection module passes the image features to a multi-scale detection head based on anchor box detection, and combines the supervision of the loss function to obtain detection and recognition results.

[0026] In one embodiment, step 3 is specifically implemented as follows: For network training, the initial learning rate is set to 1×10 -2 The SGD optimizer is used to train for 300 epochs, and the backbone network is frozen for the first 50 epochs. The momentum parameter is 0.937 and the weight decay is set to 5×10 -4 The Pytorch framework is used to build the denoising network and detection network, the binary classification loss function and GIOU loss are used to evaluate the quality of target detection, and the cosine learning rate decay strategy and Dropout strategy are adopted to optimize the model.

[0027] Compared with the existing technology, the present invention has made the following contributions in research and innovation:

[0028] 1. The denoising module designed in this invention can remove noise and extract the key feature areas in the image. The key areas are then passed to the detector to obtain the target location and classification information. The denoising module can further improve the performance of small target detection.

[0029] 2. By combining the denoising module and the detection module in the model, the method proposed in this paper achieves an accuracy of 87.39% in infrared image denoising and small target detection tasks under adverse weather conditions of rain, snow and fog. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments.

[0031] Figure 1 It is a flow chart of the specific operation process of the present invention;

[0032] Figure 2 It is the overall architecture diagram of the present invention, which includes a filter denoising module, a full-resolution denoising module and a detector module;

[0033] Figure 3 is a schematic structural diagram of the full-resolution denoising module of the present invention;

[0034] Figure 4 It is a schematic diagram of the structure of the YOLOv5 detector of the present invention;

[0035] Figure 5 This is a sample image of the three-category infrared image small target dataset of the present invention. DETAILED DESCRIPTION

[0036] The present invention will be further described below with reference to the accompanying drawings.

[0037] like Figure 1-5 As shown in Tables 1-6, the process of the infrared image denoising and small target detection method under severe weather conditions is shown. The specific steps of the method of the present invention are:

[0038] Step 1. Based on the task objectives of infrared image denoising and small target detection, infrared images suitable for small target detection and recognition were collected and organized against aerial, ground, and ocean backgrounds. The label boxes of each image in the dataset were checked using labelimg software. Missing and mislabeled targets were relabeled and saved as new annotation documents. The formats of infrared images and annotation files were unified: the image format was JPG, the annotation file format was JSON, and the image resolution was 384*288 and 640*480. Based on the above infrared images, all images were randomly divided into three equal parts and three types of noise were added: Gaussian noise, Poisson noise, and salt and pepper noise. The noise intensity was 10dB. This yielded a three-category infrared image small target dataset for the experiments of this invention. Finally, the number of classification samples, the number of targets, and the target size of the three-category infrared image small target dataset were counted.

[0039] Table 1 shows a statistical analysis of samples from each category in the noisy three-category infrared image small target dataset (GAS-Noise). The GAS-Noise dataset consists of 21,193 images with 98,612 instance targets. The dataset is divided into training and test sets in a 4:1 ratio. Objects are categorized into three categories: aerial targets, vehicle targets, and ship targets. The aerial target dataset is from the IST-A dataset, published by Xu Hai in the paper "Multi-Scale Multi-Level Residual Feature Fusion for Real-Time Infrared Small Target Detection." It contains 12,593 images, of which 12,811 are instance targets. 90% of the images in this category contain only one target, and 55% of the targets occupy only 0.02% of the image area. The vehicle target dataset consists of 5,000 images and 80,894 instance targets. The average pixel size of the targets is 40*40 pixels, and the vehicle targets are densely distributed, including many overlapping and occluded objects. The ship target consists of 3600 images and 4907 instance targets, and the average pixel size of the target is about 60*60. The datasets of vehicle targets and ship targets are from the PaddlePaddle AI website. We first unify the image format of the three collected datasets and the xml document tagging format of the target box, and then use labelimg software to check and verify the annotation box of each picture in the dataset, re-label the missed and mislabeled targets and save the new annotation document. Finally, all infrared images are divided into three equal parts and three types of noise are added: Gaussian noise, Poisson noise and salt and pepper noise, and the noise intensity is 10dB. The noise distribution is shown in Table 2, and the samples of the dataset are shown in Figure 5 .

[0040] Table 1 Noised three-category infrared image small target dataset

[0041] category Number of pictures (pieces) Target total number (units) background Average target size flying target 12,593 12,811 Sky 6*6 Vehicle Target 5,000 80,894 Ground, building 40*40 Ship Target 3,600 4,907 ocean 60*60 total 21,193 98,612 /

[0042] Table 2 Noise distribution of three-category infrared image small target dataset

[0043] category Gaussian noise (Zhang) Poisson noise (Zhang) Salt and pepper noise (photo) flying target 4198 4198 4197 Vehicle Target 1667 1667 1666 Ship Target 1200 1200 1200 total 7065 7065 7063

[0044] Step 2. Infrared images in bad weather contain more noise than infrared images in normal weather. The network framework designed by the present invention for infrared image denoising and small target detection in bad weather consists of two main components: a denoising network and a detection network. The denoising network consists of a filter denoising module for rain, snow and fog noise and a full-resolution denoising module for suppressing background noise. The filter denoising module includes a median filter, a wavelet denoising filter and a sharpening filter. The detection network uses the YOLOv5 detector. The comprehensive framework is as follows Figure 2 shown.

[0045] In the denoising network, we divide it into two submodules according to the type of noise. Noises such as rain, snow and fog generated in bad weather exist in the form of filtering on the infrared image. The present invention uses a filter denoising module to process local noise, which can also be called a local denoising module. The step of filter denoising is to pass the image through a median filter, a wavelet denoising filter and a sharpening filter in sequence to perform local denoising on the image. The median filter and the wavelet denoising filter are used to process Gaussian noise, Poisson noise and salt and pepper noise, and the sharpening filter is used to adjust the sharpness of the infrared image. The median filter replaces the value of the central pixel with the median of the neighboring pixels. It is more effective for Gaussian noise and salt and pepper noise, and can retain the edge information of the image. When the target is discrete, the formula of the median filter is:

[0046] median(x1, y1) = median(I(x1+i, y1+j)|i=0,1,...,N-1,j=0,1,...N-1)

[0047] Among them, the median (x1, y1) is the pixel value of the image after median filtering, the median represents the median of the pixel values ​​in the neighborhood, I(x1+i, y1+j) is the pixel value of the input image at position (x1+i, y1+j), and N is the size of the neighborhood. Noise can be effectively suppressed by finding the median of the neighborhood around the target pixel. Wavelet denoising is an image denoising method based on wavelet transform. Wavelet transform decomposes the image into components of different frequencies, removes high-frequency noise through threshold processing, and then performs inverse transform. The present invention uses wavelet transform to decompose and reconstruct the image, which can remove noise and retain image details. The wavelet denoising method is written as:

[0048] Wavelet denoising (x1, y1) = W -1 (T(W(I(x1,y1))))

[0049] Among them, W is the wavelet transform, T is the threshold function, which is used to perform threshold processing on the wavelet coefficients, W -1 is the inverse wavelet transform, and I(x1, y1) is the output of the image features after the median filter, which is the pixel value at (x1, y1). After the median filter and wavelet denoising filter, the image is passed to the sharpening filter to adjust the contrast between the image background and the target. The sharpening filter is used to highlight image details. The formula is as follows:

[0050] F(x1, λ)=I(x1)+λ(I(x1)-Gau(I(x1)))

[0051] Here, I(x1) is the output of the wavelet denoising filter, representing the pixel value at coordinate x1, Gau() represents a Gaussian filter, and the parameter λ is a proportional factor. This sharpening operation is differentiable with respect to both x and λ, and the degree of sharpening can be adjusted by optimizing the parameter λ to achieve better object detection performance. The above three filters form a local denoising module. The median filter and wavelet denoising filter suppress rain, snow, and fog noise in inclement weather. The sharpening filter adjusts the sharpness of the infrared image, enhancing the key target area, which facilitates the detection and recognition of small targets in infrared images.

[0052] Table 3 Detailed parameters of full-resolution denoising module

[0053]

[0054] The full-resolution denoising module adopts a structure based on convolutional neural networks, which can process background noise without changing the image feature resolution and comprehensively optimize global feature information. This module can also be called a global denoising module. The detailed parameters of the full-resolution denoising module are shown in Table 3. In traditional operations, downsampling operations are usually used for feature extraction, which can reduce network parameters and avoid gradient explosion, but will lead to the loss of contextual information. Upsampling can obtain a larger receptive field, which is beneficial to global feature extraction, but requires the addition of additional artificial information. Predicting in these erroneous pixels will lead to reduced target detection performance. Therefore, the present invention proposes a full-resolution denoising module that does not require downsampling and upsampling operations. Infrared images can be denoised without changing the image feature resolution, solving the difficulties of contextual information loss and adding invalid information. The full-resolution denoising module consists of 12 convolution-normalization-activation function blocks, 4 layers of residual convolution blocks and three jump links, such as Figure 3As shown in Figure 2. The full-resolution denoising module is divided into three stages according to the data flow order: the downsampling stage, the transfer connection stage, and the upsampling stage. The downsampling stage is used for image feature extraction: the image input passes through two layers of convolution (convolution 1 and 2) - normalization - activation function blocks, followed by convolution downsampling to obtain input feature A1. Input feature A1 passes through two layers of convolution (convolution 3 and 4) - normalization - activation function blocks, followed by convolution downsampling to obtain input feature A2. Input feature A2 passes through two layers of convolution (5 and 6) - normalization - activation function blocks, followed by convolution downsampling to obtain input feature A3. Then, the transfer connection stage begins: input feature A3 enters the 4-layer convolution residual block in the middle layer to obtain the restored feature B3. The upsampling stage is used for denoising and feature restoration. To enhance feature sharing, features are concatenated: B3 and A3 are concatenated, interpolated, and upsampled, and then pass through two layers of convolution (convolution 7 and 8) - normalization - activation function blocks to obtain restored feature B2. B2 and A2 are concatenated, interpolated, and upsampled, and then pass through two layers of convolution (convolution 9 and 10) - normalization - activation function blocks to obtain restored feature B1. B1 and A1 are concatenated, interpolated, and upsampled, and then pass through two layers of convolution (convolution 11 and 12) - normalization - activation function blocks to obtain the final full-resolution denoised image. The formula for the full-resolution denoising module is as follows:

[0055] I coarse =FRM(I noise )

[0056] Among them I noise and I coarse Represents the original noisy infrared image and the image after the full-resolution denoising module. By using L2 loss, it is ensured that the full-resolution module can obtain the denoising result without pixel distortion:

[0057]

[0058] in, is the L2 loss, I bg represents a noise-free infrared image. The full-resolution module denoises the infrared image, allowing the detector to extract accurate semantic information. We will conduct subsequent ablation experiments to verify the effectiveness of the proposed full-resolution module.

[0059] The feature information obtained by the local denoising module and the global denoising module is fused to obtain the optimal denoising feature map, which is then passed into the detection network to facilitate the detection network to identify the location information and category information of small targets in the infrared image. In the detection network, the present invention uses the YOLOv5 detector as the detection network. The YOLOv5 detector is divided into three parts: feature extraction module, feature fusion module and detection module. The detailed parameters are shown in Table 4. The incoming feature information is passed through the feature extraction module, feature fusion module and detection module in turn. The feature extraction module randomly scales and crops the image features at the input end and passes them into the backbone network containing the Focus structure and the CSP structure. The feature fusion module passes the image features into a top-down and bottom-up combined feature fusion network for fusion. The detection module passes the image features into a multi-scale detection head based on anchor frame detection, and obtains detection and recognition results in combination with the supervision of the loss function. The structure of the detector is as follows Figure 4 shown.

[0060] Table 4 Detailed parameters of the detection network

[0061]

[0062] Step 3. Based on the noisy three-category infrared image small target dataset, the framework proposed in this invention is trained to obtain a model for denoising and recognizing small targets in infrared images.

[0063] The self-built dataset consists of 21,193 images, consisting of 16,954 training images and 4,239 test images. Figure 5 The self-built three-category infrared image small target dataset is presented. The entire network is trained on 2 GPUs (NVIDIA GTX-3090). For network training, the initial learning rate is set to 1×10 -2 The SGD optimizer is used to train for 300 epochs, and the backbone network is frozen for the first 50 epochs. The momentum parameter is 0.937 and the weight decay is set to 5×10 -4 The Pytorch framework is used to build the denoising network and detection network, the binary classification loss function and GIOU loss are used to evaluate the quality of target detection, and the cosine learning rate decay strategy and Dropout strategy are adopted to optimize the model.

[0064] Dropout is a classic regularization strategy in the field of deep learning. It can reduce the overfitting problem of the model by disassociating related neurons during model training. In the present invention, we set the value of Dropout to 0.1. The reason why Dropout can effectively avoid the overfitting problem is that it can prevent some selected neurons from exchanging information with the rest of the neurons during training, so that the related neurons lose the hidden nodes associated with them, and ultimately lead to their inability to interact with the model's calculation process. Each update in the Dropout strategy process can be regarded as an iteration of different models on different subsets of the training set. It can reduce the complex interactions between many neurons, thereby effectively reducing the occurrence of overfitting problems.

[0065] Step 4. Apply the trained infrared image denoising and small object detection models to the test set for testing. Compare the detection results with those of other state-of-the-art methods to verify the model's accuracy. The accuracy of all methods was evaluated on the GAS-Noise dataset. We selected Faster RCNN, YOLOv5, YOLOv7, IA-YOLO, and DENet as benchmarks for comparison with our proposed method.

[0066] We compared our proposed method with other state-of-the-art denoising and detection models using the GAS-Noise test set. Table 5 provides a comparison of the detection accuracy and mean average precision (MAP) for each class of different denoising and detection models on the GAS-Noise test set. Each detector uses a 640×640 pixel resolution input. As shown in Table 5, our proposed method achieves a mean average precision (mAP) of 87.39% at an Intersection over Union (IoU) threshold of 0.5. This method outperforms Faster R-CNN, YOLOv5, YOLOv7, IA-YOLO, and DENet by 29.29%, 3.01%, 2.8%, 3.02%, and 10.98%, respectively. For aerial targets, our proposed method achieves a 4.95% improvement over the next best method. For vehicle detection, our proposed method achieves a 0.64% improvement over the next best method. For ship detection, our proposed method achieves a 0.79% improvement over the next best method. Experiments demonstrate that our proposed method not only enhances the overall detection performance of the detection system but also demonstrates its effectiveness in infrared image denoising.

[0067] Table 5 Comparison of detection accuracy of the present invention and other state-of-the-art methods

[0068] method AP1 AP2 AP3 mAP FasterRCNN 7.20％ 70.54％ 91.34％ 58.85％ YOLOv5 69.25％ 90.31％ 95.82％ 85.12％ YOLOv7 66.86％ 92.57％ 96.55％ 85.33％ IA-YOLO 66.66％ 92.45％ 96.22％ 85.11％ DENet 44.68％ 91.52％ 95.24％ 77.15％ Ours 73.84％ 93.21％ 97.34％ 88.13％

[0069] Step 5. Ablation experiments were conducted on the denoising networks of the infrared image denoising and small object detection models. The ablation results are detailed in Table 6. The experimental results described in Table 6 include ablation comparisons of four combinations: the base module, only the filter denoising module, only the full-resolution denoising module, and both the filter and full-resolution denoising modules. YOLOv5 was used as the detector for all of these. When the filter denoising module and the full-resolution denoising module were integrated into the detector, the mean average performance (mAP) increased by 1.05% and 1.5%, respectively. The detection accuracy of aerial objects increased by 1.89% and 2.31%, respectively. The detection accuracy of vehicle objects increased by 0.82% and 1.53%, respectively. The detection accuracy of ship objects increased by 0.41% and 0.64%, respectively. When both the filter denoising module and the full-resolution denoising module were added, the mAP increased by 4.59%, the detection accuracy of aerial objects increased by 4.59%, the detection accuracy of vehicle objects increased by 2.9%, and the detection accuracy of ship objects increased by 1.52%. These experiments highlight the importance of jointly training the local and global denoising modules.

[0070] Table 6 Comparison of ablation experiments on the test set of the present invention on the noisy dataset

[0071] method AP1 AP2 AP3 mAP YOLOv5 69.25％ 90.31％ 95.82％ 85.12％ Filtering and denoising 71.14％ 91.13％ 96.23％ 86.17％ Full-resolution denoising 71.56％ 91.84％ 96.46％ 86.62％ Full denoising network 73.84％ 93.21％ 97.34％ 88.13％

[0072] Step 6. Analysis and summary of experimental results.

[0073] The code implementation of this invention is based on the PyTorch framework, running on Ubuntu 20.04, using two NVIDIA TiTANRTX graphics cards for parallel acceleration. After training, we compared the results with test results. Our model suppresses noise and improves the detection accuracy of small infrared targets in inclement weather conditions such as rain, snow, and fog. By combining denoising and detection tasks in an end-to-end model, this model addresses the inherent challenges of infrared image denoising and small target detection in inclement weather. This model incorporates a filter denoising module for local denoising to suppress rain, snow, and fog noise in inclement weather conditions, and a full-resolution denoising module for global denoising to suppress background noise in infrared images, enhancing the generalization of target feature information. This model combines the denoising network with the YOLOv5 detection network to achieve infrared image denoising and small target detection and recognition in inclement weather conditions. Experimental results show that our model achieves an average accuracy of 87.39% on a noisy dataset, demonstrating significant performance improvements in infrared image denoising and small target detection in inclement weather conditions.

[0074] The above description is a further detailed description of the present invention in conjunction with specific / preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art of the present invention may make various substitutions or modifications to the described embodiments without departing from the scope of the present invention, and such substitutions or modifications should be considered to fall within the scope of protection of the present invention.

[0075] Parts of the present invention that are not described in detail belong to the common knowledge of those skilled in the art.

Claims

1. A method for infrared image denoising and small target detection in severe weather conditions, characterized in that: The steps are as follows: Step 1. Based on the task objectives of infrared image denoising and detection and recognition, a three-category infrared image small target dataset was constructed. The dataset was randomly divided into three equal parts and three types of noise were added: Gaussian noise, Poisson noise, and salt and pepper noise. The noise intensity was 10 dB. Step 2. Build a network architecture for infrared image denoising and small target detection in adverse weather conditions. The overall framework includes a filter denoising module, a full-resolution denoising module, and a YOLOv5 detector. The detector uses a loss supervision function to improve the detection performance of small targets. Step 3. Based on the noisy three-category infrared image small target dataset, the proposed network architecture is trained to obtain a model for infrared image denoising and recognition. Step 4. Test the trained infrared image denoising and small target detection model on the test set to verify the model's AP and mAP indicators; Noise generated by bad weather such as rain, snow, and fog exists in the form of filtering on infrared images. The filter denoising module is used to process local noise, which can also be called a local denoising module. The filter denoising step is to pass the image through the median filter, wavelet denoising filter, and sharpening filter in sequence to perform local denoising on the image. The median filter and wavelet denoising filter are used to process Gaussian noise, Poisson noise, and salt and pepper noise. The sharpening filter is used to adjust the sharpness of the infrared image. When the target is discrete, the formula of the median filter is: median(x1,y1)=median(I(x1+i,y1+j)|i=0,1,...,N-1,j=0,1,...N-1) Among them, the median (x1, y1) is the pixel value of the image after median filtering, the median represents the median of the pixel values ​​in the neighborhood, I(x1+i, y1+j) is the pixel value of the input image at position (x1+i, y1+j), and N is the size of the neighborhood; wavelet denoising is an image denoising method based on wavelet transform; wavelet transform decomposes the image into components of different frequencies, removes high-frequency noise through threshold processing, and then performs inverse transform; wavelet transform is used to decompose and reconstruct the image. The wavelet denoising method is written as: Wavelet denoising (x1, y1) = W -1 (T(W(I(x1,y1)))) Among them, W is the wavelet transform, T is the threshold function, which is used to perform threshold processing on the wavelet coefficients, W -1 is the inverse wavelet transform, I(x1,y1) is the output of the image feature after the median filter, which is the pixel value at (x1,y1); after the median filter and wavelet denoising filter, the image is passed to the sharpening filter to adjust the contrast between the image background and the target; the sharpening filter is used to highlight the image details, and the formula is as follows: F(x1,λ)=I(x1)+λ(I(x1)-Gau(I(x1))) Among them, I(x1) is the output of the wavelet denoising filter, which represents the pixel value at coordinate x1, Gau() represents the Gaussian filter, and the parameter λ is the positive proportional factor. This sharpening operation is differentiable with respect to both x and λ, and the degree of sharpening can be adjusted by optimizing the parameter λ to obtain better object detection performance. The above three filters constitute a local denoising module, which suppresses rain, snow and fog noise in bad weather through the median filter and wavelet denoising filter, and adjusts the sharpness of the infrared image through the sharpening filter, enhancing the key area of ​​the target, which is conducive to the detection and recognition of small targets in the infrared image. The full-resolution denoising module consists of 12 convolution-normalization-activation function blocks, 4 layers of residual convolution blocks and three skip links. The full-resolution denoising module is divided into three stages according to the data flow order, namely the downsampling stage, the transfer connection stage and the upsampling stage. The downsampling stage is used for image feature extraction: the image input passes through two layers of convolution-normalization-activation function blocks and then undergoes convolution downsampling to obtain input feature A1. The input feature A1 passes through two layers of convolution-normalization-activation function blocks and then undergoes convolution downsampling to obtain input feature A2. The input feature A2 passes through two layers of convolution-normalization-activation function blocks and then undergoes convolution downsampling to obtain input feature A3. Then Entering the transfer connection stage: input feature A3 enters the 4-layer convolution residual block of the middle layer to obtain the restored feature B3; the upsampling stage is used for denoising and feature restoration. In order to strengthen feature sharing, feature splicing will be performed: B3 and A3 are spliced ​​and then interpolated and upsampled, and then pass through two layers of convolution-normalization-activation function blocks in sequence to obtain the restored feature B2. B2 and A2 are spliced ​​and then interpolated and upsampled, and then pass through two layers of convolution-normalization-activation function blocks in sequence to obtain the restored feature B1. B1 and A1 are spliced ​​and then interpolated and upsampled, and then pass through two layers of convolution-normalization-activation function blocks in sequence to obtain the final full-resolution denoised image; the formula of the full-resolution denoising module is as follows: I coarse =FRM(I noise ) Among them I noise and I coarse Represents the original noisy infrared image and the image after the full-resolution denoising module; by using L2 loss, it is ensured that the full-resolution module can obtain the denoising result without pixel distortion: in, is the L2 loss, I bg Represents a noise-free infrared image.

2. The infrared image denoising and small target detection method for severe weather conditions according to claim 1, characterized in that: Step 1: According to the task objectives of infrared image denoising and small target detection, infrared images suitable for small target detection and recognition tasks are collected and sorted under aerial background, ground background and ocean background respectively. The labelimg software is used to check the annotation box of each image in the dataset, and the missed and mislabeled targets are re-labeled and saved as new annotation documents. The formats of infrared images and annotation files are unified. The image format is JPG, and the annotation file format is JSON. The image resolution is 384*288 and 640*480. Based on the above infrared images, all images are randomly divided into three equal parts and three types of noise, Gaussian noise, Poisson noise and salt and pepper noise, are added. The noise intensity is 10dB, and the noisy three-category infrared image small target dataset used for the experiment of the present invention is obtained. Finally, the number of classification samples, the number of targets and the target size of the noisy three-category infrared image small target dataset are counted.

3. The infrared image denoising and small target detection method for use in severe weather conditions according to claim 1 or 2, characterized in that: The feature information obtained by the local denoising module and the global denoising module is fused to obtain the optimal denoising feature map, which is then passed into the detection network to facilitate the detection network to identify the location information and category information of small targets in the infrared image. In the detection network, the YOLOv5 detector is used as the detection network. The YOLOv5 detector is divided into three parts: feature extraction module, feature fusion module and detection module. The incoming feature information passes through the feature extraction module, feature fusion module and detection module in sequence. The feature extraction module randomly scales and crops the image features at the input end and passes them into the backbone network containing the Focus structure and CSP structure. The feature fusion module passes the image features into a top-down and bottom-up feature fusion network for fusion; the detection module passes the image features into a multi-scale detection head based on anchor box detection, and obtains the detection and recognition results in combination with the supervision of the loss function.

4. The infrared image denoising and small target detection method for severe weather conditions according to claim 3, wherein: Step 3 is implemented as follows: For network training, the initial learning rate is set to 1×10 -2 The SGD optimizer is used to train for 300 epochs, and the backbone network is frozen for the first 50 epochs; the momentum parameter is 0.937, and the weight decay is set to 5×10 -4 ; The Pytorch framework is used to build the denoising network and detection network, the binary classification loss function and GIOU loss are used to evaluate the quality of target detection, and the cosine learning rate decay strategy and Dropout strategy are adopted to optimize the model.

Images