Infrared image denoising method based on two-level feature fusion

Abstract

The present invention discloses an infrared image denoising method based on two-stage feature fusion. The specific steps are as follows: first stage: (1) extracting the initial features of the infrared noise image; (2) inputting the initial features into the first encoding structure to obtain encoding features of three different scales; (3) inputting the encoding features of the last scale into the first decoding structure to obtain decoding features with enhanced details; (4) inputting the decoding features of the last scale into the self-supervised attention module. Second stage: (5) extracting the initial features of the infrared noise image again and splicing them with the features extracted by the self-supervised attention module; (6) adjusting the number of feature channels of the spliced ​​features, inputting them into the second encoding structure, and fusing them with the features of the corresponding scales in steps 2 and 3; (7) inputting the fused features of the last scale into the second decoding structure to obtain decoding features with enhanced details; (8) reconstructing and restoring a clear infrared image.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular relates to an infrared image denoising method based on two-level feature fusion. Background Art

[0002] Infrared radiation is essentially a type of electromagnetic radiation with a wavelength between visible light and microwaves. Its short wavelength is adjacent to the red wavelength of visible light, while its long wavelength borders microwaves. Medium- and long-wave infrared (MWIR) primarily detects the infrared spectrum emitted by a target object, which is affected by multiple factors, including the target's characteristics and the application scenario. Due to issues with MWIR detector manufacturing processes and quantization efficiency, the production of high-density, small-pixel infrared detector devices remains difficult and expensive. The structural organization of the detector array and current technological levels cannot achieve a 100% fill factor. Furthermore, according to the Nyquist sampling theorem, the sampling frequency of the detector array is limited. It is difficult to achieve twice the Nyquist frequency of natural scene images. Undersampling in MWIR images can lead to signal aliasing, resulting in blurred infrared images and low spatial resolution. The main sources of noise in infrared images include 1 / f noise, shot noise, dark current noise, and readout noise. Furthermore, due to uncertainties in detector and operational amplifier manufacturing processes, fringe and non-uniformity noise are particularly noticeable in infrared images.

[0003] Traditional image denoising algorithms are often limited by their limited versatility and time-consuming nature. However, with the recent increase in computing power, researchers have been able to achieve scientific breakthroughs by processing vast amounts of data. In deep learning denoising tasks, previous single-stage network structures often suffer from semantic loss and blurred edges when reconstructing images. Multi-stage network structures, which mostly employ cascade operations and encoder-decoder structures, are effective at encoding semantic information but fail to preserve spatial image detail. Therefore, given the low resolution, complex noise sources, and poor image detail of infrared images, how to more effectively restore edge and texture details without losing spatial structure is a crucial issue in image denoising. Summary of the Invention

[0004] To address the problems of existing infrared image denoising methods, the present invention proposes a method for infrared image denoising based on two-stage feature fusion. This method employs a two-stage network strategy: the first-stage network extracts semantic information, and the second-stage network fuses this extracted semantic information to reconstruct an image with both spatial structure and detail information. Furthermore, an improved G residual block is used as a feature extraction module, resulting in a better reconstruction quality.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] The infrared image denoising method based on two-level feature fusion includes the following steps:

[0007] First level:

[0008] Step 1: Using the first feature extraction convolutional layer to extract the initial features of the infrared image polluted by noise;

[0009] Step 2: Input the extracted initial features into the first encoding structure. During the encoding process, three different scale encoding features are obtained by continuous downsampling, and each scale corresponds to a resolution.

[0010] In step 3, the encoded features at the last scale are input into the first decoding structure. The resolution is restored through continuous upsampling to obtain the decoded features at the three different scales. At the same scale, the encoded features extracted in step 2 are added to the decoded features using a skip-layer connection to obtain the detail-enhanced decoded features.

[0011] Step 4: Input the detail-enhanced decoded features of the last scale outputted in step 3 into the self-supervised attention module, which generates the denoised reconstructed image X1 and extracts the attention-enhanced feature representation.

[0012] Level 2:

[0013] Step 5: Use the second feature extraction convolution layer to extract the initial features of the infrared image in step 1, and perform a splicing operation with the attention-enhanced feature representation described in step 4;

[0014] Step 6: The concatenated features are subjected to a feature adjustment convolution layer to adjust the number of feature channels and are input into a second encoding structure. The second encoding structure has the same structure as the first encoding structure but different parameters. The features of the corresponding scales of the first-level network encoding and decoding parts are added to the second encoding features at the current scale obtained after each downsampling process after convolution to obtain the fused second encoding features at the current scale.

[0015] Step 7: Input the second encoding features fused in step 6 into a second decoding structure. The second decoding structure has the same structure as the first decoding structure but different parameters. Skip-layer connections are also used to obtain detail-enhanced decoding features.

[0016] In step 8, the detail-enhanced decoded features outputted in step 7 are subjected to a reconstruction convolutional layer to obtain the denoised reconstructed image X2 of this level, i.e., the restored clear infrared image.

[0017] Furthermore, in step 2, the G residual block and the 2×2 downsampling layer are used as a basic downsampling module. During the downsampling process, the 2×2 downsampling module is used twice to double the number of feature channels and obtain coding features of three different scales.

[0018] Furthermore, the processing steps of the G residual block are: after the input passes through a 3×3 convolution layer, the group normalization method GN is used to normalize the parameters according to the channel correlation, and then the Gaussian error linear unit is used for nonlinear processing.

[0019] Furthermore, in step 3, the first decoding structure uses a transposed convolution layer with a convolution kernel size of 4; after upsampling, the features are mapped using a residual connection block; in addition, a skip layer connection is added at each scale to compensate for the information loss caused by resampling.

[0020] Furthermore, in step 4, the self-supervised attention module includes the following calculation steps:

[0021] Step 41: Input features Use residual reconstruction convolution layer to generate residual image Here H×W represents the spatial dimension and C represents the number of channels;

[0022] Step 42: Add the residual image and the input noise image to obtain a restored clear image.

[0023] Step 43, reconstruct the image X1 through the attention convolution layer and Sigmoid function to generate a mask The mask is used for attention enhancement;

[0024] Step 44, initial feature F in After being processed by the feature map convolution layer, it is multiplied by the mask and then multiplied by the feature F in Added together, this generates a series of attention-enhancing features;

[0025] In step 45, the features obtained in step 44 are concatenated with the initial features extracted in the second stage, thereby completing the function of partially enhancing attention.

[0026] The present invention designs a two-stage feature fusion denoising network, introduces a self-supervised attention module, and uses a G residual block as a feature extraction module to implement an infrared image denoising method that retains edge and texture detail information. The beneficial effects of the method of the present invention are: (1) because the entire model is a fully convolutional neural network, image denoising can be performed on infrared image inputs of any scale without setting additional parameters; (2) through the two-stage feature fusion network design, the use of a self-supervised attention module, and the G residual block as a feature extraction module, the infrared image's edge and texture detail information can be retained while restoring the infrared image's spatial structure information. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a structural block diagram of the implementation of the method of the present invention;

[0028] Figure 2 Flow chart of the method of the present invention;

[0029] Figure 3 Schematic diagram of the specific structure of the self-supervised attention module in an embodiment of the present invention;

[0030] Figure 4 Schematic diagram of the specific structure of the G residual block and the basic residual block in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] Reference Figure 1 The infrared image denoising method based on two-stage feature fusion in this embodiment employs a hierarchical feature extraction and reconstruction architecture. The second stage fuses the first-stage attention-enhanced features and the multi-scale features of each encoding / decoding structure to produce a reconstructed image. The contaminated infrared image is used as input and cropped to a 128×128 image block. The method specifically includes the following steps:

[0033] First level:

[0034] Step 1, first feature extraction: Use a 3×3 first feature extraction convolutional layer with a bias unit to extract the initial features of the image. The number of channels of the input infrared noise image is 1, and the number of extracted feature channels is set to 32. H and W are the height and width of the image.

[0035] Step 2, the first encoding structure: use the G residual block and the 2×2 downsampling layer as a basic downsampling module. The G residual block passes the input through a 3×3 feature map convolution layer, and then uses the group normalization method GN to normalize the parameters according to the channel correlation. Then, it uses the Gaussian error linear unit GELU for nonlinear processing, and then repeats a 3×3 convolution and Gaussian error linear unit operation. The output is added to the original input after the 1×1 convolution dimension transformation, which is the residual connection. In the downsampling process, two 2×2 downsampling modules are used to double the number of feature channels, obtaining features of three different scales, each scale corresponding to a resolution.

[0036] Step 3, first decoding structure: the upsampling layer uses a transposed convolution layer with a convolution kernel size of 4; after upsampling, the features are mapped using a basic residual connection block, where the basic residual connection block is the G residual block without the group normalization layer; in addition, a skip layer connection is added at each scale (different from the residual connection of each basic block), and the encoded features extracted in step 2 are added to the decoded features to obtain detail-enhanced decoded features to compensate for the information loss caused by resampling.

[0037] Step 4, Self-supervised Attention Enhancement Module SAM: The features output by the first decoding stage Use a 3×3 residual reconstruction convolution layer to generate a residual image Here H×W represents the spatial dimension and C represents the number of channels. Then the residual image and the input noisy image are added to obtain the restored clear image. Then X1 passes through a 3×3 attention convolution layer and a Sigmoid function to generate a mask. These masks play a role of attention enhancement; the initial feature F in After a convolutional layer, it is multiplied by the mask and then by the feature F in When added together, a series of attention-enhanced features will be generated, with a feature channel count of 32. These features will be concatenated with the initial features extracted at the next level to complete part of the attention enhancement function.

[0038] Level 2:

[0039] Step 5, second feature extraction: A 3×3 second feature extraction convolution layer is used to extract the initial features of the same noise-contaminated infrared image, and then concatenated with the attention-enhanced feature representation extracted by the previous level self-supervised attention enhancement module SAM. The number of feature channels becomes 64, and then a 3×3 feature adjustment convolution layer is used to adjust the feature channel dimension to 32.

[0040] Step 6, Second Encoding Structure: The features after adjusting the feature channel dimension are input into the second encoding structure. The second encoding structure has the same structure as the first encoding structure but different parameters. After each downsampling process, the second encoding features at that scale obtained from the encoding and decoding parts of the first-level network are added to the features at the corresponding scale through a 3×3 convolution to obtain the fused second encoding features at that scale.

[0041] Step 7, second decoding structure: Input the second encoding features of the last scale after fusion in step 6 into the second decoding structure. The second decoding structure has the same structure as the first decoding structure but different parameters. It also uses skip-layer connections to obtain detail-enhanced decoding features.

[0042] Step 8, reconstruction: Use a 3×3 reconstruction convolution layer to obtain the denoised reconstructed image X2 of this level, that is, the restored clear infrared image.

[0043] The loss function used in this embodiment when training the two-stage feature fusion denoising network is That is, the L1 loss between the final reconstruction result of the Sth level and the original image. In this embodiment, the image block size used in training is 128128, the width of the first layer feature extraction of the network is set to 32, and the learning rate is preset to 2×10 -4 , and the cosine annealing strategy is used to gradually decay it to 1×10 -6 The Adam optimizer was used, with β1 and β2 set to 0.9 and 0.999, respectively. The loss function used was Equation 3.8. The entire network was trained in a Linux-based environment running BasicSR 1.2.0, Pytorch 1.8.0 with cu111, and TorchVision 0.9.0 with cu111. Distributed training was performed using two RTX 3090 GPUs, with a batch size of 16 on each GPU and a total of 4 × 105 iterations. A single training session took approximately three days to complete.

Claims

1. Infrared image denoising method based on two-level feature fusion, characterized by: The method comprises the following steps: First level: Step 1: Using the first feature extraction convolutional layer to extract the initial features of the infrared image polluted by noise; Step 2: Input the extracted initial features into the first encoding structure. During the encoding process, three different scale encoding features are obtained by continuous downsampling, and each scale corresponds to a resolution. In step 3, the encoding features of the last scale are input into the first decoding structure. The resolution is restored through continuous upsampling to obtain the decoding features of the three different scales mentioned above. At the same scale, the encoding features extracted in step 2 are added to the decoding features using skip-layer connections to obtain the detail-enhanced decoding features. Step 4: Input the detail-enhanced decoded features of the last scale outputted in step 3 into the self-supervised attention module, which generates the denoised reconstructed image X1 and extracts the feature representation of the attention enhancement. Level 2: Step 5: Use the second feature extraction convolution layer to extract the initial features of the infrared image in step 1, and perform a splicing operation with the attention-enhanced feature representation described in step 4; Step 6: The concatenated features are subjected to a feature adjustment convolution layer to adjust the feature channel dimension, and the features after the feature channel dimension adjustment are input into a second coding structure. The second coding structure has the same structure as the first coding structure but different parameters. In addition, the features of the corresponding scales of the first-level network encoding and decoding parts are added to the second coding features at the current scale obtained after each downsampling process after convolution processing to obtain the fused second coding features at the current scale. Step 7: Input the second encoding features fused in step 6 into a second decoding structure. The second decoding structure has the same structure as the first decoding structure but different parameters. Skip-layer connections are also used to obtain detail-enhanced decoding features. In step 8, the detail-enhanced decoded features output in step 7 are reconstructed using a convolutional layer to obtain the denoised reconstructed image X2 of this level, i.e., the restored clear infrared image.

2. The infrared image denoising method based on two-level feature fusion according to claim 1 is characterized in that: In step 2, the G residual block and the 2×2 downsampling layer are used as a basic downsampling module. During the downsampling process, the 2×2 downsampling module is used twice to double the number of feature channels and obtain coding features of three different scales.

3. The infrared image denoising method based on two-level feature fusion according to claim 2 is characterized in that: The processing steps of the G residual block are: after the input passes through a 3×3 convolution layer, the group normalization method GN is used to normalize the parameters according to the channel correlation, and then the Gaussian error linear unit is used for nonlinear processing.

4. The infrared image denoising method based on two-level feature fusion according to claim 1, characterized in that: In step 3, the first decoding structure uses a transposed convolution layer with a convolution kernel size of 4; after upsampling, the residual connection block is used to map the features; In addition, a skip connection is added at each scale to compensate for the information loss caused by resampling.

5. The infrared image denoising method based on two-level feature fusion according to claim 1, characterized in that: In step 4, the self-supervised attention module includes the following calculation steps: Step 41: Input features Use residual reconstruction convolution layer to generate residual image Here H×W represents the spatial dimension and C represents the number of channels; Step 42: Add the residual image and the input noise image to obtain a restored clear image. Step 43, reconstruct the image X1 through the attention convolution layer and Sigmoid function to generate a mask The mask is used for attention enhancement; Step 44, initial feature F in After being processed by the feature map convolution layer, it is multiplied by the mask and then multiplied by the feature F in Added together, this generates a series of attention-enhancing features; In step 45, the features obtained in step 44 are concatenated with the initial features extracted in the second stage, thereby completing the function of partially enhancing attention.

Images