An image compound noise denoising method of a double-encoder branch U-shaped structure network

Abstract

The application discloses a kind of image composite noise denoising methods of double-encoder branch U-shaped structure network, belong to image processing technical field.The method for realizing of the present application is: the characteristic scale difference of two kinds of noise components is analyzed, and the U-shaped network structure with two encoder branches is constructed.U-shaped network structure includes two different scale encoder branches, a decoder branch and feature fusion connection module.The encoder branch of conventional scale attention is used to learn Gaussian noise characteristics, and the encoder branch of enlarged scale attention is used to learn stripe noise characteristics.The feature fusion module is used to fuse the features obtained by the two encoder branches, to realize the fusion of Gaussian noise and stripe noise characteristics.The decoder branch obtains the depth feature map provided by the encoder branch and the fusion feature map provided by the feature fusion module.Combined with upsampling operation, Gaussian random noise and space-time domain row-column stripe noise are gradually removed, the denoised image is restored, and the image quality of imaging instrument is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to a method for image denoising, in particular to a method for removing the superimposed complex noise of Gaussian noise and stripe noise of imaging instruments, which is suitable for application in the fields of infrared imaging devices, ultraviolet imaging devices, remote sensing imaging technology and the like, and belongs to the technical field of image processing. BACKGROUND

[0002] Imaging instruments such as infrared cameras, ultraviolet cameras and remote sensing cameras are prone to image quality affected by noise due to limited device manufacturing process level, high manufacturing difficulty level and limited sensor yield, resulting in insufficient signal-to-noise ratio, which seriously restricts their expansion application in various application scenarios.

[0003] At present, the main noise components in the image of the imaging instrument include Gaussian random noise and space-time domain row-column stripe noise. The Gaussian random noise is randomly distributed in the image spatial domain, and a conventional local receptive field convolutional neural network can achieve good learning effect. The space-time domain stripe noise usually spans the entire image, showing randomness of row-column position and randomness of gray scale distribution. At present, the traditional image processing algorithm is mainly used for stripe position estimation and removal. Due to the non-locality of stripe noise, it is difficult to achieve good denoising effect by using a neural network with a small receptive field.

[0004] Early image denoising technology of imaging instruments mainly focuses on the removal of stripe noise, and adopts methods such as spatial filter method, transform domain filter method and noise characteristic modeling removal method. In recent years, deep learning methods have made significant progress in image denoising. For example, the U-shaped network structure based on convolutional neural network can learn noise features and remove complex noise. However, due to the limited receptive field of CNN network, the noise removal capability is still limited. Although the introduction of Transformer technology and self-attention mechanism improves the performance of image denoising, these methods are mainly designed for visible light images and do not fully consider the special case of mixed Gaussian noise and stripe noise in imaging instrument images.

[0005] Therefore, it is of great significance to develop a method that can effectively remove Gaussian noise and stripe noise in imaging instrument images to improve the image quality and application range of imaging instruments. This method needs to consider the feature differences of the two kinds of noise and can maintain good denoising effect in different application scenarios. SUMMARY

[0006] Aiming at the problems of Gaussian random noise and row and column stripe noise commonly existing in imaging instruments, the purpose of the present application is to provide a kind of image composite noise denoising method of double encoder branch U-shaped structure network, based on double encoder branch, U-shaped network structure is constructed, based on U-shaped network structure, Gaussian random noise and space-time domain row and column stripe noise are removed, that is, image composite noise denoising is realized, and the image quality of imaging instrument is improved.The present application is suitable for the application in the fields of infrared imaging device, ultraviolet imaging device and remote sensing imaging technology.

[0007] The purpose of the present application is realized by the following technical solutions:

[0008] The present application discloses a kind of image composite noise denoising method of double encoder branch U-shaped structure network, by analyzing the characteristic scale difference of two kinds of noise components, U-shaped network structure with two encoder branches is constructed.U-shaped network structure includes two different scale encoder branches, a decoder branch and feature fusion connection module.Two different scale encoder branches are respectively basic scale attention encoder (BSA_encoder), enlarge scale attention encoder (ESA_encoder) of regular scale attention.The basic scale attention encoder is used to learn Gaussian noise feature, and the enlarge scale attention encoder is used to learn stripe noise feature.The feature fusion module is used to fuse the features obtained by two encoder branches.The decoder branch obtains the deep feature map provided by the encoder branch and the fusion feature map provided by the feature fusion module.Combined with up-sampling operation, Gaussian random noise and space-time domain row and column stripe noise are gradually removed, image signal-to-noise ratio is improved, denoised image is restored, and the image quality of imaging instrument is improved.

[0009] The present application discloses a kind of image composite noise denoising method of double encoder branch U-shaped structure network, including the following steps:

[0010] S1, a sequence of noise images is collected, and a noise image and true value image data pair is established.The collection mode of a single set of noise image and true value image pair is that, for a static scene, M sequence noise images are collected, one of which is randomly selected as a noise image A1, then the average value of the M sequence noise images is obtained, to obtain a true value image B1, then the noise image A1 and the true value image B1 form a set of noise-true value image pair.Continue to collect noise image sequence until N sets of noise-true value image pairs used for neural network training are obtained.

[0011] S2, by analyzing the characteristic scale difference of the two noise components, a U-shaped network structure with two encoder branches is constructed. The U-shaped network structure includes two different scale encoder branches, a decoder branch and a feature fusion module. The two different scale encoder branches are a regular scale attention encoder branch and an expanded scale attention encoder branch. The regular scale attention encoder branch is used to learn Gaussian noise features, and the expanded scale attention encoder branch is used to learn stripe noise features. The feature fusion module is used to fuse the features obtained by the two encoder branches. The decoder branch obtains the deep feature map provided by the encoder branch and the fusion feature map provided by the feature fusion module, and then up-samples to generate.

[0012] The regular scale attention encoder branch includes at least one regular scale attention layer and a down-sampling layer, which is used to input a noise image and mainly learns the features of Gaussian noise by using the local feature learning ability of the convolutional neural network. The feature processing of the i-th layer of the regular scale attention encoder branch can be represented as:

[0013] L i+1 =D i (C BSA_i (L i ))

[0014] Wherein, L i represents the feature map transmitted by the upper encoder layer, L i+1 represents the feature map output by the current encoder layer, C BSA_i represents the regular scale convolutional network of the current regular scale attention encoder layer, D i represents the down-sampling operation adopted by the current encoder layer.

[0015] The expanded scale attention encoder branch includes at least one expanded scale attention layer and a down-sampling layer. The expanded scale attention layer is constructed based on a large-size convolution kernel of a convolutional neural network or a Transformer network. By using the feature learning ability of the expanded scale attention convolutional neural network or the non-local attention Transformer network, the row and column stripe noise across the entire image is learned. The feature processing of the i-th layer of the expanded scale attention encoder branch can be represented as:

[0016]

[0017] Wherein, represents the feature map transmitted by the upper encoder layer, represents the feature map output by the current encoder layer, represents the expanded scale convolutional network or Transformer network of the current expanded scale attention encoder layer, D i represents the down-sampling operation adopted by the current encoder layer.

[0018] The feature fusion module is constructed based on a convolutional neural network, and the noise feature maps obtained by the regular scale attention layer and the enlarged scale attention layer are fused through the channel fusion and feature transformation capability of the convolutional neural network, and the feature map tensor scale is adjusted to the required scale of the decoder branch.

[0019] The decoder branch includes at least one regular scale attention layer and an up-sampling layer, and the feature decoding and feature reorganization are performed based on the regular scale convolutional neural network, and the image detail information is recovered step by step through the up-sampling, and finally the denoised image is recovered. The feature processing of the i-th layer of the regular scale attention decoder branch can be represented as:

[0020]

[0021] wherein, represents the feature map transmitted from the lower decoder layer, represents the feature map output by the current decoder layer, represents the regular scale convolutional network of the current regular scale attention decoder layer, U i represents the up-sampling operation adopted by the current encoder layer.

[0022] S3, using the noise-true value image pair obtained in step S1 as a data set for training the neural network built in S2. The training process is as follows: loading N groups of noise-true value image pairs into a training data loader Dataloader, and grouping the data according to the training requirements; inputting the grouped data into the double-encoder U-shaped network according to the group, and performing multi-level feature extraction and down-sampling by the two encoder branches to generate high-dimensional feature maps, and then performing feature reconstruction and up-sampling by the decoder branch step by step to recover the image features and generate the prediction results; using the back propagation algorithm to optimize the network parameters, and minimizing the loss function between the output and the true value image; iterating the training until the network converges, and obtaining the trained U-shaped network structure for image denoising.

[0023] S4, inputting the processed noise image into the U-shaped network structure trained in step S3, and performing feature extraction and dimension reduction on the input image by the double-encoder branch of the U-shaped network, processing the extracted features by the middle layer of the network, learning to remove Gaussian random noise and space-time domain row and column stripe noise, combining the up-sampling operation by the decoder branch, gradually recovering the spatial resolution of the image, obtaining the denoised image, improving the signal-to-noise ratio of the image, and improving the image quality of the imaging instrument, i.e. realizing image composite noise denoising based on the double-encoder branch U-shaped structure network.

[0024] Advantages:

[0025] 1. The application discloses a double-encoder branch U-shaped network structure image composite noise denoising method, which analyzes the characteristic scale difference of two noise components, and constructs a U-shaped network structure with two encoder branches. The two different scale encoder branches are a regular scale attention encoder branch and an enlarged scale attention encoder branch. The regular scale attention encoder branch is used to learn Gaussian noise features, and the enlarged scale attention encoder branch is used to learn stripe noise features, so that good composite noise removal effect can be achieved, and the image signal-to-noise ratio is improved.

[0026] 2. The application discloses a double-encoder branch U-shaped network structure image composite noise denoising method, and the U-shaped network structure includes two different scale encoder branches, a decoder branch and a feature fusion connection module. The feature fusion connection module fuses the noise features of different scales obtained by the two encoder branches in the feature space and the channel, realizes effective fusion of Gaussian noise and stripe noise features, facilitates further processing of the decoder branch, and improves the image composite noise denoising precision and efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 A flowchart of the U-shaped network structure based on the double-encoder branch for denoising proposed by the application.

[0028] Figure 2 The U-shaped network structure of the double-encoder branch proposed by the application.

[0029] Figure 3 An effect diagram for denoising of an infrared image in the embodiment of the application. DETAILED DESCRIPTION

[0030] In order to better illustrate the purposes and advantages of the application, the content of the application will be further described below in combination with the drawings and examples.

[0031] Example 1

[0032] As shown in the drawings, Figure 1 The application discloses a double-encoder branch U-shaped network structure image composite noise denoising method, and the specific implementation steps are as follows:

[0033] S1, a noise image sequence is collected, and a noise image and a true value image data pair are established. The collection mode of a single set of noise image and true value image pair is that 200 sequence noise images are collected for a static scene, one of which is randomly selected as a noise image A1, and then the average value of the 200 sequence noise images is obtained to obtain a true value image B1, so that the noise image A1 and the true value image B1 form a set of noise-true value image pair. Continue to collect the noise image sequence until 800 sets of noise-true value image pairs for neural network training are obtained.

[0034] S2, by analyzing the characteristic scale difference of two noise components, a U-shaped network structure with two encoder branches is constructed. The U-shaped network structure includes two different scale encoder branches, a decoder branch and a feature fusion module. The two different scale encoder branches are a regular scale attention encoder branch and an expanded scale attention encoder branch. The regular scale attention encoder branch is used to learn Gaussian noise features, and the expanded scale attention encoder branch is used to learn stripe noise features. The feature fusion module is used to fuse the features obtained by the two encoder branches. The decoder branch obtains the deep feature map provided by the encoder branch and the fusion feature map provided by the feature fusion module, and then up-samples to generate.

[0035] The regular scale attention encoder branch includes three layers of regular scale attention layers and a down-sampling layer, which is used for inputting a noise image and mainly learning the features of Gaussian noise by means of the local feature learning ability of the convolutional neural network. The feature processing of the i-th layer of the regular scale attention encoder branch can be represented as:

[0036] L i+1 =D i (C BSA_i (L i ))

[0037] Wherein, L i represents the feature map transmitted from the upper encoder layer, L i+1 represents the feature map output by the current encoder layer, C BSA_i represents the regular scale convolutional network of the current regular scale attention encoder layer, which is built based on a 3x3 size convolution kernel, D i represents the maximum value pooling down-sampling operation adopted by the current encoder layer.

[0038] The expanded scale attention encoder branch includes three layers of expanded scale attention layers and a down-sampling layer, which is based on a 7x7 size large size convolution kernel of the convolutional neural network, and learns the row and column stripe noise across the whole image by means of the expanded scale attention convolutional neural network feature learning ability. The feature processing of the i-th layer of the expanded scale attention encoder branch can be represented as:

[0039]

[0040] Wherein, represents the feature map transmitted from the upper encoder layer, represents the feature map output by the current encoder layer, represents the expanded scale convolutional network of the current expanded scale attention encoder layer, which is built based on a 7x7 size convolution kernel, D i represents the maximum value pooling down-sampling operation adopted by the current encoder layer.

[0041] The feature fusion module is constructed based on a convolutional neural network, noise feature maps obtained by the conventional scale attention layer and the enlarged scale attention layer are fused through the channel fusion and feature transformation capability of the convolutional neural network, and the feature map tensor scale is adjusted to the required scale of the decoder branch.

[0042] The decoder branch includes three conventional scale attention layers and an up-sampling layer, and features are decoded and reorganized through a convolutional neural network with a 3*3 size convolution kernel, and image detail information is recovered through linear up-sampling at each level, and finally a denoised image is recovered. The feature processing of the i-th layer of the conventional scale attention decoder branch can be represented as:

[0043]

[0044] wherein, represents the feature map transmitted from the lower decoder layer, represents the feature map output by the current decoder layer, represents the conventional scale convolution network of the current conventional scale attention decoder layer, which is built based on a 3*3 size convolution kernel, and U i represents a linear up-sampling method used in the current encoder layer.

[0045] S3, using the noise-true value image pair obtained in step S1 as a data set for training the neural network built in S2. The training process is as follows: 800 groups of noise-true value image pairs are loaded into the training data loader Dataloader, and the data is grouped according to the training requirements; the grouped data is input into the double-encoder U-shaped network, and the two encoder branches perform multi-level feature extraction and down-sampling to generate high-dimensional feature maps, which are then processed by the decoder branch for feature reconstruction and up-sampling to recover the image features and generate the prediction results; the network parameters are optimized using the back propagation algorithm to minimize the loss function between the output and the true value image; the network is iteratively trained until it converges, and a trained U-shaped network structure for image denoising is obtained.

[0046] S4, input the processed noise image into the U-shaped network structure trained in step S3, the double-encoder branch of the U-shaped network extracts and reduces the dimension of the input image, the middle layer of the network processes the extracted features, learns to remove Gaussian random noise and spatial and temporal domain stripe noise, and the decoder branch combines the up-sampling operation to gradually recover the spatial resolution of the image, obtains a denoised image, and improves the signal-to-noise ratio of the image and the image quality of the imaging instrument.

[0047] Through the above steps, the Gaussian noise and stripe noise in the image of the imaging instrument are effectively removed, and the image quality is significantly improved. This method can ensure the denoising effect while being suitable for real-time processing in practical applications.

[0048] The above detailed description of the specific description, the purpose, technical scheme and beneficial effects of the application are further described in detail, it should be understood that the above description is only a specific embodiment of the application, and is not used to limit the protection scope of the application, any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the application should be included in the protection scope of the application.

Claims

1. A method for image compound noise denoising of a dual-encoder branch U-shaped structure network, characterized in that: Comprising the following steps, S1, collecting a sequence of noise images, and establishing a noise image and a true value image data pair; the collection mode of a single set of noise image and true value image pair is that, for a static scene, M sequence noise images are collected, one of which is randomly selected as a noise image A1, then the average value of the M sequence noise images is obtained to obtain a true value image B1, and the noise image A1 and the true value image B1 form a set of noise-true value image pairs; continue to collect a sequence of noise images until N sets of noise-true value image pairs for neural network training are obtained; S2, by analyzing the characteristic scale difference of the two noise components, a U-shaped structure network with two encoder branches is constructed; the U-shaped structure network includes two different scale encoder branches, a decoder branch and a feature fusion module; the two different scale encoder branches are a regular scale attention encoder branch and an expanded scale attention encoder branch; The regular scale attention encoder branch is used for learning Gaussian noise features, and the expanded scale attention encoder branch is used for learning stripe noise features; The feature fusion module is used for fusing the features obtained by the two encoder branches; The decoder branch obtains the deep feature map provided by the encoder branch and the fusion feature map provided by the feature fusion module, and then generates by upsampling; S3, using the noise-true value image pairs obtained in step S1 as a data set for training the neural network built in S2; the training process is as follows: load the N sets of noise-true value image pairs into a training data loader Dataloader, and group the data according to the training requirements; input the grouped data into the double-encoder U-shaped structure network, the two encoder branches perform multi-level feature extraction and down sampling to generate high-dimensional feature maps, and then the decoder branch performs feature reconstruction and upsampling step by step to restore the image features and generate the prediction results; Use the back propagation algorithm to optimize the network parameters, minimize the loss function between the output and the true value image; iterate until the network converges to obtain the trained U-shaped structure network for image denoising; S4, input the processed noise image into the U-shaped structure network trained in step S3, the double-encoder branch part of the U-shaped structure network performs feature extraction and dimension reduction on the input image, the middle layer of the network processes the extracted features, learns to remove Gaussian random noise and space-time domain row and column stripe noise, and the decoder branch combines the upsampling operation to gradually restore the spatial resolution of the image to obtain the denoised image, that is, the double-encoder branch U-shaped structure network realizes image composite noise denoising.

2. The image compound noise denoising method of a dual-encoder branch U-shaped structure network according to claim 1, characterized in that: In step S2, The regular scale attention encoder branch includes at least one regular scale attention layer and a down sampling layer, which is used for inputting the noise image and learning the features of the Gaussian noise by means of the local feature learning ability of the convolutional neural network; The feature processing of the i-th layer of the regular scale attention encoder branch is represented as: L i+1 = D i (C BSA_i (L i )) wherein L i represents the feature map passed in from the upper encoder layer, L i+1 represents the feature map output by the current encoder layer, C BSA_i represents the regular scale convolutional network of the current regular scale attention encoder layer, D i represents the down-sampling operation adopted by the current encoder layer.

3. The image compound noise denoising method of a dual-encoder branch U-shaped structure network according to claim 2, characterized in that: The enlarged-scale attention encoder branch comprises at least one enlarged-scale attention layer and a down-sampling layer, the enlarged-scale attention layer is constructed based on a convolutional neural network with a large-size convolution kernel or a Transformer network, the feature learning capability of the convolutional neural network with enlarged-scale attention or the Transformer network with non-local attention is used to learn the row and column stripe noise across the whole image; The feature processing of the i-th layer of the enlarged-scale attention encoder branch can be expressed as: wherein, represents the feature map passed in from the upper encoder layer, represents the feature map output by the current encoder layer, represents the upscaling convolutional network or the Transformer network of the current upscaling attention encoder layer, D i represents the down-sampling operation adopted by the current encoder layer.

4. The image compound noise denoising method of a dual-encoder branch U-shaped structure network according to claim 3, characterized in that: The feature fusion module is constructed based on a convolutional neural network, the noise feature maps obtained by the regular-scale attention layer and the enlarged-scale attention layer are fused through the channel fusion and feature transformation capability of the convolutional neural network, and the feature map tensor scale is adjusted to the required scale of the decoder branch.

5. The image compound noise denoising method of a dual-encoder branch U-shaped structure network according to claim 4, characterized in that: The decoder branch comprises at least one regular-scale attention layer and an up-sampling layer, the feature decoding and feature recombination are performed based on the regular-scale convolutional neural network, and the image detail information is recovered step by step through up-sampling, and finally the denoised image is recovered; The feature processing of the i-th layer of the regular-scale attention decoder branch can be expressed as: wherein, denotes the feature maps passed in from the lower decoder layer, denotes the feature maps output by the current decoder layer, denotes the regular scale convolutional network of the current regular scale attention decoder layer, U i denotes the up-sampling operation employed by the current encoder layer.

Images