Image composite noise reduction method of dual-encoder branch U-shaped structure network
Through the dual encoder branch U-shaped structure network, the characteristics of Gaussian noise and fringe noise in imaging instruments are learned and removed, which solves the problem of insufficient signal-to-noise ratio and significantly improves image quality and application range.
Patent Information
- Application Number
- CN202410997350.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-24
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-07-24
AI Technical Summary
The prior art is difficult to effectively remove the composite noise of Gaussian noise and fringe noise in imaging instruments, resulting in insufficient signal-to-noise ratio, limiting the expansion of imaging instruments in application scenarios.
Using a dual encoder branch U-shaped structure network, Gaussian noise and stripe noise characteristics are learned through the attention encoder branches of conventional and expanded scales, and upsampling and denoising through feature fusion module and decoder branches to improve the image signal-to-noise ratio .
实现了对成像仪器图像中高斯噪声和条纹噪声的有效去除,显著提高了图像质量和信噪比,适用于红外、紫外和遥感成像技术。
Smart Images

Figure CN119090756A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an image denoising method, and in particular to a method for removing composite noise superimposed by Gaussian noise and stripe noise of an imaging instrument. The method is suitable for applications in the fields of infrared imaging devices, ultraviolet imaging devices, remote sensing imaging technology, etc., and belongs to the field of image processing technology. Background Art
[0002] Imaging instruments such as infrared cameras, ultraviolet cameras, and remote sensing cameras have limited device manufacturing process levels, high manufacturing difficulty levels, and limited sensor output. The image quality is easily affected by noise, resulting in insufficient signal-to-noise ratio, which seriously restricts their expanded application in various application scenarios.
[0003] At present, the main noise components in imaging instrument images include Gaussian random noise and spatiotemporal row and column stripe noise. Gaussian random noise is randomly distributed in the spatial domain of the image, and a good learning effect can usually be achieved by using a conventional convolutional neural network with a local receptive field. Spatiotemporal stripe noise usually spans the entire image, showing randomness in row and column positions and randomness in grayscale distribution. Currently, stripe position estimation and removal are mainly based on traditional image processing algorithms. Due to the non-locality of stripe noise, it is difficult to achieve a good denoising effect using a neural network with a smaller receptive field.
[0004] Early imaging instrument image denoising technology focused on the removal of streak noise, using methods based on spatial filters, transform domain filtering, and targeted modeling and removal based on noise characteristics. In recent years, deep learning methods have made significant progress in the field of image denoising. For example, the U-shaped network structure based on convolutional neural networks can learn noise features and remove complex noise. However, due to the limited receptive field of CNN networks, the ability to remove noise is still limited. Although the introduction of Transformer technology and self-attention mechanism has improved the performance of image denoising, these methods are mainly designed for visible light images and do not fully consider the special case of mixed Gaussian noise and streak noise in imaging instrument images.
[0005] Therefore, developing a method that can effectively remove Gaussian noise and stripe noise from imaging instrument images is of great significance for improving the image quality and application scope of imaging instruments. This method needs to consider the characteristic differences of the two types of noise at the same time and be able to maintain good denoising effects in different application scenarios. Summary of the invention
[0006] Aiming at the problem that imaging instruments usually have Gaussian random noise and row and column stripe noise, the purpose of the present invention is to provide an image composite noise denoising method of a dual encoder branch U-shaped structure network, construct a U-shaped network structure based on dual encoder branches, remove Gaussian random noise and spatiotemporal row and column stripe noise based on the U-shaped network structure, that is, realize image composite noise denoising and improve the image quality of imaging instruments. The present invention is suitable for applications in the fields of infrared imaging devices, ultraviolet imaging devices, and remote sensing imaging technology.
[0007] The objective of the present invention is achieved through the following technical solutions:
[0008] The present invention discloses an image composite noise denoising method of a dual encoder branch U-shaped structure network. By analyzing the characteristic scale difference of two noise components, a U-shaped network structure with two encoder branches is constructed. The U-shaped network structure includes two encoder branches of different scales, a decoder branch and a feature fusion connection module. The two encoder branches of different scales are respectively a basic scale attention encoder branch (BSA_encoder) and an enlarged scale attention encoder branch (ESA_encoder). The encoder branch of the conventional scale attention is used to learn Gaussian noise features, and the encoder branch of the enlarged scale attention is used to learn stripe noise features. The feature fusion module is used to fuse the features obtained by the two encoder branches. The decoder branch obtains the depth feature map provided by the encoder branch and the fusion feature map provided by the feature fusion module. Combined with the upsampling operation, Gaussian random noise and spatiotemporal row and column stripe noise are gradually removed, the image signal-to-noise ratio is improved, the denoised image is restored, and the image quality of the imaging instrument is improved.
[0009] The present invention discloses a method for denoising image composite noise using a dual encoder branch U-shaped structure network, comprising the following steps:
[0010] S1. Collect a noise image sequence and establish a noise image and true value image data pair. The collection method of a single set of noise image and true value image pairs is to collect M sequence noise images for a static scene, randomly select one of them as the noise image A1, and then take the average of the M sequence noise images to obtain the true value image B1. The noise image A1 and the true value image B1 form a set of noise-true value image pairs. Continue to collect noise image sequences until N sets of noise-true value image pairs are obtained for neural network training.
[0011] S2. By analyzing the difference in feature scales of the two noise components, a U-shaped network structure with two encoder branches is constructed. The U-shaped network structure includes two encoder branches of different scales, a decoder branch, and a feature fusion module. The two encoder branches of different scales are the encoder branch of conventional scale attention and the encoder branch of enlarged scale attention. The encoder branch of conventional scale attention is used to learn Gaussian noise features, and the encoder branch of enlarged scale attention is used to learn stripe noise features. The feature fusion module is used to fuse the features obtained by the two encoder branches. The decoder branch obtains the deep feature map provided by the encoder branch and the fused feature map provided by the feature fusion module, and then upsamples and generates.
[0012] The conventional scale attention encoder branch includes at least one conventional scale attention layer and a downsampling layer, which is used to input a noisy image and mainly learn the characteristics of Gaussian noise with the help of the local feature learning ability of the convolutional neural network. The feature processing of the i-th layer of the conventional scale attention encoder branch can be expressed as:
[0013] L i+1 =D i (C BSA_i (L i ))
[0014] Among them, L i Represents the feature map passed in by the upper encoder layer, L i+1 Represents the feature map output by the current encoder layer, C BSA_i represents the normal-scale convolutional network of the current normal-scale attention encoder layer, D i Indicates the downsampling operation adopted by the current encoder layer.
[0015] The upscaled attention encoder branch includes at least one upscaled attention layer and a downsampling layer, and the upscaled attention layer is constructed based on a convolutional neural network with a large-size convolution kernel or a Transformer network. With the help of the feature learning ability of the upscaled attention convolutional neural network or the non-local attention Transformer network, the row and column stripe noise across the rows and columns of the entire image is learned. The feature processing of the i-th layer of the upscaled attention encoder branch can be expressed as:
[0016]
[0017] in, Represents the feature map passed in by the upper encoder layer, represents the feature map output by the current encoder layer, represents the upscaled convolutional network or Transformer network of the current upscaled attention encoder layer, D i Indicates the downsampling operation adopted by the current encoder layer.
[0018] The feature fusion module is built based on a convolutional neural network. Through the channel fusion and feature transformation capabilities of the convolutional neural network, the noise feature maps obtained by the conventional scale attention layer and the enlarged scale attention layer are fused, and the feature map tensor scale is adjusted to the scale required by the decoder branch.
[0019] The decoder branch includes at least one normal-scale attention layer and an upsampling layer, performs feature decoding and feature reorganization based on a normal-scale convolutional neural network, and restores image detail information step by step through upsampling, and finally restores the denoised image. The feature processing of the i-th layer of the normal-scale attention decoder branch can be expressed as:
[0020]
[0021] in, represents the feature map passed in by the lower decoder layer, represents the feature map output by the current decoder layer, represents the normal-scale convolutional network of the current normal-scale attention decoder layer, U i Indicates the upsampling operation used by the current encoder layer.
[0022] S3. Use the noise-true value image pairs obtained in step S1 as a data set for the neural network training built in S2. The training process is as follows: load N groups of noise-true value image pairs into the training data loader Dataloader, and group the data according to the training requirements; input the grouped data into the dual encoder U-type network in groups, and the two encoder branches perform multi-level feature extraction and downsampling to generate high-dimensional feature maps, and then perform feature reconstruction and upsampling step by step through the decoder branch to restore the image features and generate prediction results; use the back propagation algorithm to optimize the network parameters and minimize the loss function between the output and the true value image; iterate the training until the network converges to obtain a trained U-type network structure for image denoising.
[0023] S4. Input the processed noisy image into the U-shaped network structure trained in step S3. The dual encoder branch of the U-shaped network extracts features and reduces the dimension of the input image. The middle layer of the network processes the extracted features and learns to remove Gaussian random noise and row and column stripe noise in the spatiotemporal domain. The decoder branch combines the upsampling operation to gradually restore the spatial resolution of the image to obtain a denoised image, thereby improving the image signal-to-noise ratio and the image quality of the imaging instrument. That is, image composite noise denoising is achieved based on the dual encoder branch U-shaped structure network.
[0024] Beneficial effects:
[0025] 1. The present invention discloses an image composite noise denoising method of a dual encoder branch U-shaped structure network. By analyzing the characteristic scale difference of two noise components, a U-shaped network structure with two encoder branches is constructed. The two encoder branches of different scales are respectively an encoder branch of conventional scale attention and an encoder branch of enlarged scale attention. The encoder branch of conventional scale attention is used to learn Gaussian noise characteristics, and the encoder branch of enlarged scale attention is used to learn stripe noise characteristics, which can achieve good composite noise removal effect and improve image signal-to-noise ratio.
[0026] 2. The present invention discloses a method for denoising composite noise of an image using a dual encoder branch U-shaped structure network. The U-shaped network structure includes two encoder branches of different scales, a decoder branch and a feature fusion connection module. The feature fusion connection module performs feature space fusion and channel fusion on noise features of different scales acquired by the two encoder branches, realizes effective fusion of Gaussian noise and stripe noise features, facilitates further processing of the decoder branch, and improves the accuracy and efficiency of denoising composite noise of the image. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 This is a flow chart of denoising based on the U-shaped structure network with dual encoder branches proposed in the present invention.
[0028] Figure 2 This is a U-shaped network structure with dual encoder branches proposed by the present invention.
[0029] Figure 3 This is a diagram showing the effect of infrared image denoising in an embodiment of the present invention. DETAILED DESCRIPTION
[0030] In order to better illustrate the purpose and advantages of the present invention, the invention is further described below with reference to the accompanying drawings and examples.
[0031] Embodiment 1:
[0032] like Figure 1 As shown, this embodiment discloses an image composite noise denoising method of a dual encoder branch U-shaped structure network, and the specific implementation steps are as follows:
[0033] S1. Collect a noise image sequence and establish a noise image and true value image data pair. The collection method of a single set of noise image and true value image pairs is to collect 200 sequence noise images for a static scene, randomly select one of them as the noise image A1, and then take the average of the 200 sequence noise images to obtain the true value image B1. The noise image A1 and the true value image B1 form a set of noise-true value image pairs. Continue to collect noise image sequences until 800 sets of noise-true value image pairs are obtained for neural network training.
[0034] S2. By analyzing the difference in feature scales of the two noise components, a U-shaped network structure with two encoder branches is constructed. The U-shaped network structure includes two encoder branches of different scales, a decoder branch, and a feature fusion module. The two encoder branches of different scales are the encoder branch of conventional scale attention and the encoder branch of enlarged scale attention. The encoder branch of conventional scale attention is used to learn Gaussian noise features, and the encoder branch of enlarged scale attention is used to learn stripe noise features. The feature fusion module is used to fuse the features obtained by the two encoder branches. The decoder branch obtains the deep feature map provided by the encoder branch and the fused feature map provided by the feature fusion module, and then upsamples and generates.
[0035] The conventional scale attention encoder branch includes three layers of conventional scale attention layers and downsampling layers, which are used to input noisy images and mainly learn the characteristics of Gaussian noise with the help of the local feature learning ability of the convolutional neural network. The feature processing of the i-th layer of the conventional scale attention encoder branch can be expressed as:
[0036] L i+1 =D i (C BSA_i (L i ))
[0037] Among them, L i Represents the feature map passed in by the upper encoder layer, L i+1 Represents the feature map output by the current encoder layer, C BSA_i represents the normal-scale convolutional network of the current normal-scale attention encoder layer, built based on a 3×3 convolution kernel, D i Indicates the maximum pooling downsampling adopted by the current encoder layer.
[0038] The upscaled attention encoder branch includes three upscaled attention layers and downsampling layers, based on a convolutional neural network with a large convolution kernel of 7×7 size, and with the help of the feature learning ability of the convolutional neural network of upscaled attention, it learns the row and column stripe noise across the rows and columns of the entire image. The feature processing of the i-th layer of the upscaled attention encoder branch can be expressed as:
[0039]
[0040] in, Represents the feature map passed in by the upper encoder layer, represents the feature map output by the current encoder layer, represents the upscaled convolutional network of the current upscaled attention encoder layer, which is built based on a 7×7 convolution kernel. i Indicates the maximum pooling downsampling operation used by the current encoder layer.
[0041] The feature fusion module is built based on a convolutional neural network. Through the channel fusion and feature transformation capabilities of the convolutional neural network, the noise feature maps obtained by the conventional scale attention layer and the enlarged scale attention layer are fused, and the feature map tensor scale is adjusted to the scale required by the decoder branch.
[0042] The decoder branch includes three regular scale attention layers and upsampling layers, performs feature decoding and feature reorganization based on a convolutional neural network with a 3×3 convolution kernel, and restores image detail information step by step through linear upsampling, and finally restores the denoised image. The feature processing of the i-th layer of the regular scale attention decoder branch can be expressed as:
[0043]
[0044] in, represents the feature map passed in by the lower decoder layer, represents the feature map output by the current decoder layer, represents the normal-scale convolutional network of the current normal-scale attention decoder layer, which is built based on a 3×3 convolution kernel. i Indicates the linear upsampling method used by the current encoder layer.
[0045] S3. Use the noise-true image pairs obtained in step S1 as a data set for the neural network training built in S2. The training process is as follows: load 800 sets of noise-true image pairs into the training data loader Dataloader, and group the data according to the training requirements; input the grouped data into the dual encoder U-type network in groups, and the two encoder branches perform multi-level feature extraction and downsampling to generate high-dimensional feature maps, and then perform feature reconstruction and upsampling step by step through the decoder branch to restore the image features and generate prediction results; use the back propagation algorithm to optimize the network parameters and minimize the loss function between the output and the true value image; iterate the training until the network converges to obtain a trained U-type network structure for image denoising.
[0046] S4. Input the processed noisy image into the U-shaped network structure trained in step S3. The dual encoder branch of the U-shaped network extracts features and reduces the dimension of the input image. The middle layer of the network processes the extracted features and learns to remove Gaussian random noise and row and column stripe noise in the spatiotemporal domain. The decoder branch combines the upsampling operation to gradually restore the spatial resolution of the image to obtain a denoised image, thereby improving the image signal-to-noise ratio and the image quality of the imaging instrument.
[0047] Through the above steps, the present invention realizes the effective removal of Gaussian noise and stripe noise in the image of the imaging instrument, and significantly improves the image quality. While ensuring the denoising effect, the method is suitable for real-time processing in practical applications.
[0048] The specific description above further illustrates the purpose, technical solutions and beneficial effects of the invention in detail. It should be understood that the above is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A method for denoising image composite noise using a dual encoder branch U-shaped structure network, characterized in that: The following steps are included: S1, collecting a noise image sequence, and establishing a noise image and a true value image data pair; the collection method of a single set of noise image and true value image pairs is to collect M sequence noise images for a static scene, randomly select one of them as a noise image A1, and then take an average value of the M sequence noise images to obtain a true value image B1, so that the noise image A1 and the true value image B1 form a set of noise-true value image pairs; continue to collect noise image sequences until N sets of noise-true value image pairs for neural network training are obtained; S2. By analyzing the feature scale differences of the two noise components, a U-shaped network structure with two encoder branches is constructed; the U-shaped network structure includes two encoder branches of different scales, a decoder branch and a feature fusion module; the two encoder branches of different scales are respectively an encoder branch of conventional scale attention and an encoder branch of enlarged scale attention; The encoder branch of the conventional scale attention is used to learn Gaussian noise features, and the encoder branch of the enlarged scale attention is used to learn stripe noise features; The feature fusion module is used to fuse the features obtained by the two encoder branches; The decoder branch obtains the deep feature map provided by the encoder branch and the fused feature map provided by the feature fusion module, and then upsamples and generates; S3, using the noise-true value image pairs obtained in step S1 as a data set for the neural network training built in S2; the training process is specifically as follows: N groups of noise-true value image pairs are loaded into the training data loader Dataloader, and the data are grouped according to the training requirements; the grouped data are input into the dual encoder U-type network in groups, and the two encoder branches perform multi-level feature extraction and downsampling to generate a high-dimensional feature map, and then the decoder branch performs feature reconstruction and upsampling step by step to restore the image features and generate a prediction result; Use the back propagation algorithm to optimize the network parameters and minimize the loss function between the output and the true value image; iterate the training until the network converges to obtain a trained U-shaped network structure for image denoising; S4. Input the processed noisy image into the U-shaped network structure trained in step S3. The dual encoder branch of the U-shaped network extracts features and reduces the dimension of the input image. The middle layer of the network processes the extracted features and learns to remove Gaussian random noise and row and column stripe noise in the spatiotemporal domain. The decoder branch combines the upsampling operation to gradually restore the spatial resolution of the image and obtain the denoised image. That is, image composite noise denoising is achieved based on the dual encoder branch U-shaped structure network.
2. The image composite noise denoising method of a dual encoder branch U-shaped structure network according to claim 1, characterized in that: In step S2, The conventional scale attention encoder branch includes at least one conventional scale attention layer and a downsampling layer, which is used to input a noisy image and mainly learn the characteristics of Gaussian noise by means of the local feature learning ability of the convolutional neural network; The feature processing of the i-th layer of the conventional scale attention encoder branch is expressed as: L i+1 =D i (C BSA_i (L i )) Among them, L i Represents the feature map passed in by the upper encoder layer, L i+1 Represents the feature map output by the current encoder layer, C BSA_i represents the normal-scale convolutional network of the current normal-scale attention encoder layer, D i Indicates the downsampling operation adopted by the current encoder layer.
3. The image composite noise denoising method of a dual encoder branch U-shaped structure network as claimed in claim 2, characterized in that: The upscaled attention encoder branch includes at least one upscaled attention layer and a downsampling layer, and the upscaled attention layer is constructed based on a convolutional neural network with a large-size convolution kernel or based on a Transformer network, and the row and column stripe noise across the rows and columns of the entire image is learned by using the feature learning ability of the convolutional neural network with upscaled attention or the Transformer network with non-local attention; The feature processing of the i-th layer of the upscaled attention encoder branch can be expressed as: in, Represents the feature map passed in by the upper encoder layer, represents the feature map output by the current encoder layer, represents the upscaled convolutional network or Transformer network of the current upscaled attention encoder layer, D i Indicates the downsampling operation adopted by the current encoder layer.
4. The image composite noise denoising method of a dual encoder branch U-shaped structure network as claimed in claim 3, characterized in that: The feature fusion module is built based on a convolutional neural network. Through the channel fusion and feature transformation capabilities of the convolutional neural network, the noise feature maps obtained by the conventional scale attention layer and the enlarged scale attention layer are fused, and the feature map tensor scale is adjusted to the scale required by the decoder branch.
5. The image composite noise denoising method of a dual encoder branch U-shaped structure network as claimed in claim 4, characterized in that: The decoder branch includes at least one conventional scale attention layer and an upsampling layer, performs feature decoding and feature reorganization based on a conventional scale convolutional neural network, and restores image detail information step by step through upsampling, and finally restores the denoised image; The feature processing of the i-th layer of the conventional scale attention decoder branch can be expressed as: in, represents the feature map passed in by the lower decoder layer, represents the feature map output by the current decoder layer, represents the normal-scale convolutional network of the current normal-scale attention decoder layer, U i Indicates the upsampling operation used by the current encoder layer.
Citation Information
Patent Citations
Image fusion denoising method and system based on U-Net and pyramid module
CN112801909A
Multi-fusion comparison network based on full-dimensional dynamic convolution
CN116757955A
A method of denoising dental images through domain adaptation
US20230196515A1
Channel attention-based swin-transformer image denoising method and system
US20240193723A1
Cited By
Image processing method and device and storage medium
CN122222865A
