An image denoising method based on asymmetric structure network

Abstract

The present invention discloses an image denoising method based on an asymmetric structure network, which belongs to the fields of image processing, computer vision, etc., and mainly includes four steps: 1) collecting and processing a data set; 2) constructing an asymmetric image denoising network; 3) selecting a loss function and training the network; 4) testing new data. For the core denoising network in the method, the present invention adopts a novel asymmetric structure, which mainly includes an encoder based on Swin Transformer and a decoder based on residual convolution. This design can greatly reduce the model's computational cost while ensuring denoising performance. At the same time, the present invention also uses the RBF kernel function to improve the self-attention mechanism of Swin Transformer, further improving the model's denoising performance.

Description

Technical Field

[0001] The present invention belongs to the fields of image processing, computer vision, etc. and particularly relates to an image denoising method based on an asymmetric structure network. Background Art

[0002] With the widespread adoption of digital imaging devices, the demand for high-quality images is increasing. In real-world production and everyday life, imaging often contains various noises due to factors such as operating skill, lighting conditions, and object motion. This noise significantly impacts the practical value of images in production and everyday life. Therefore, restoring high-quality images from noisy images is of significant research interest and holds great promise for practical application.

[0003] Over the past few decades, a large number of denoising methods based on image priors have emerged, such as BM3D, which relies on self-similarity. These methods have demonstrated excellent performance on synthetic noisy images, but their performance on real noisy images is less than satisfactory. Furthermore, the high complexity of these methods limits their application in practical scenarios.

[0004] With the rapid development of deep neural networks, image denoising methods based on deep neural networks have become mainstream. Compared to traditional denoising methods, these methods eliminate the need for manual feature design, improving their adaptability. Furthermore, deep neural networks can effectively organize larger-capacity models, enabling better approximation of complex noise distributions. This type of denoising method can be traced back to the denoising autoencoder (DAE), proposed in 2008. The DAE utilizes a simple three-layer fully connected network. Its overall structure can be decomposed into an encoder and a decoder. The encoder encodes and compresses the input image, discarding redundant noise information, and the decoder then restores the clean original image. This structure has been adopted by many subsequent works, such as HiNet and Uformer. Currently, the core components of encoders and decoders mainly include two basic modules: convolutional modules and Transformer modules.

[0005] Comparing these two modules, convolution introduces the prior of locality and parameter sharing in its structure, which makes the denoising network based on pure convolution very efficient in computation. For the Transformer module, its core self-attention mechanism discards the limitations of locality and parameter sharing and minimizes the prior information as much as possible. Therefore, the Transformer-type denoising method has a larger model capacity and better denoising performance, but compared with convolution, the Transformer also needs to bear a higher computational cost.

[0006] The denoising method proposed in this paper combines the advantages of the convolutional module and the Transformer to design an asymmetric image denoising network. This asymmetric network not only ensures good denoising performance, but also greatly reduces the computational complexity of the denoising network.

[0007] References

[0008] 1.Liu Z, Lin Y, Cao Y, et al.Swin Transformer: Hierarchical VisionTransformer using Shifted Windows[C].IEEE International Conference onComputer Vision.2021:10012-10022.

[0009] 2.He K, Zhang X, Ren S, et al. Deep Residual Learning for ImageRecognition[C]. IEEE Conference on Computer Vision and PatternRecognition. 2016: 770-778. Summary of the Invention

[0010] The purpose of the present invention is to provide a computationally efficient and high-performance denoising method for real image noise. Current denoising methods based on deep neural networks mainly focus on improving denoising performance or reducing model complexity. This results in networks with strong denoising performance often being accompanied by high computational costs, while computationally efficient denoising networks sacrifice certain denoising performance. To address this issue, the present invention designs a denoising network with an asymmetric structure, which can strike a good balance between the two, greatly reducing the computational complexity of the denoising network while ensuring denoising performance.

[0011] Specific innovations: This method first designs an asymmetric encoder-decoder network for image denoising. This asymmetric image denoising network uses an encoder based on a large-capacity Swin Transformer and a computationally efficient residual convolutional decoder, enabling the denoising network to balance denoising performance and computational efficiency. Secondly, this method also improves the self-attention mechanism of the Swin Transformer in the encoder. The new RBF attention mechanism can further optimize the processing capabilities of visual data, thereby improving the denoising performance of the entire network.

[0012] The specific method steps of the present invention are:

[0013] S1. Collect images containing real noise and their corresponding noise-free images, preprocess the collected data, and then divide them into training and test sets;

[0014] S2. Construct an asymmetric image denoising network. The network adopts a codec structure, where the encoder is mainly constructed by a large-capacity Swin Transformer module [1], and the decoder is mainly composed of computationally efficient residual convolution [2]. The network specifically consists of five parts, which are: input projection layer P in , encoder E, bottleneck layer B, decoder D, output projection layer P out . Where a) input projection layer P in It consists of a convolution layer with a convolution kernel size of 3×3 and a LeakyReLU activation layer; b) The encoder E consists of four consecutive coding sub-blocks {E1, E2, E3, E4}, each coding sub-block E i Contains a feature conversion module and a downsampling module; c) The bottleneck layer B contains a feature conversion module; d) The decoder D consists of four serial decoding sub-blocks {D4, D3, D2, D1}, each decoding sub-block D i Contains an upsampling module and a feature conversion module; e) Output projection layer P out It consists of a convolution layer with a convolution kernel size of 3×3. These five parts are organized together in series, and each coding sub-block E i Decoding sub-block D of the same level i There is a skip link for the transfer of information;

[0015] S3. Construct a loss function and then use the training set to optimize the image denoising network. After each round of training, calculate the PSNR value of the network on the test set. Finally, save the parameter with the highest PSNR for subsequent inference testing.

[0016] S4. Test new data by inputting the new image into the trained image denoising network for inference, and then post-processing the network output to obtain the denoised image.

[0017] In the above method, the preprocessing described in step S1 is to crop the collected image pairs, cut the high-resolution image into multiple image slices of 256×256 size at equal intervals, and then convert the numerical range of the sliced ​​images from [0, 255] to [-1, +1]. The conversion rules are as follows:

[0018]

[0019] Where T in is the image value before conversion, T outis the converted image value.

[0020] In the above method, the feature conversion module described in step S2 b) is a SwinTransformer module using the RBF attention mechanism, which includes a Transformer that does not use an offset window and a Transformer that uses an offset window, where the size of each window is set to 8×8; secondly, the downsampling module in b) is a convolutional layer with a stride of 2 and a convolution kernel size of 4×4.

[0021] In the above method, the feature conversion module described in step S2 c) is a global Transformer module, which specifically flattens the input features along the spatial dimension, then uses the Transformer to model the relationship between the flattened features, and finally reconverts the output features to the same size as the input features.

[0022] In the above method, the upsampling module described in step S2 d) includes a neighboring interpolation upsampling layer with a sampling factor of 2 and a convolution layer with a convolution kernel size of 3×3; the feature conversion module in d) includes two convolution layers with a convolution kernel size of 3×3 and an identity link, and each convolution layer is followed by a LeakyReLU activation layer.

[0023] In the above method, the loss function described in step S3 is the Charbonnier loss function:

[0024]

[0025] Where I is the input image, is the corresponding noise-free image, ||*|| represents the L2 norm, f is the denoising network, θ is the parameter of the denoising network, and ε is 1×10 -6 .

[0026] In the above method, the post-processing described in step S4 is to re-quantize the output image, transforming the numerical range of the output image from [-1, +1] to [0, 255]. The calculation rules are as follows:

[0027]

[0028] Where T in Output image value for the network, T out is the post-processed value, and round is the rounding operation.

[0029] In the above method, the attention mechanism of Swin Transformer used in step S2 b) is RBF attention mechanism, and its calculation process is as follows: the input feature vector Projection is performed, where n is the length of the feature and d represents the dimension of each feature. After projection, three projection matrices are obtained: Where Q = Z in W Q ,K=Z in W K ,V=Z in W V , the expression of RBF attention mechanism is:

[0030]

[0031] in is the output feature vector of the RBF attention mechanism. The norm(*) operation performs L2 normalization on the row vector of the input matrix. λ is a hyperparameter with a default value of 0.02. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is the overall flow chart of an image denoising method based on an asymmetric structure network.

[0033] Figure 2 Schematic diagram of an image denoising network with an asymmetric structure.

[0034] Figure 3 This is the calculation process of the RBF attention mechanism.

[0035] Figure 4 It is the feature conversion module of the decoding sub-block. DETAILED DESCRIPTION

[0036] The technical solution of this embodiment is as follows:

[0037] Step S1. Collect the dataset and download the open-source benchmark datasets SIDD and PolyU for real-world image noise removal. Then, crop the collected high-resolution images into 256×256 image slices at equal intervals. Then, transform the data range of the sliced ​​images from [0, 255] to [-1, +1]. Finally, divide 90% of the processed data into the training set and the rest into the test set.

[0038] Step S2. Build an asymmetric denoising network. The network mainly consists of five parts, which are: input projection layer P in , encoder E, bottleneck layer B, decoder D, output projection layer P out The forward calculation process of a batch of image data through the denoising network is as follows:

[0039] a) Take a batch of images Where H×W is the spatial resolution, 3 is the number of channels, and N is the size of the batch. First, I passes through an input projection layer P in Generating low-level visual features Among them, channel C is 32;

[0040] b) The low-level visual feature X0 is then passed through four encoding sub-blocks (E i , i=1,2,3,4) serial encoders, each encoding sub-block consists of a feature conversion module and a downsampling module, where the feature conversion module is a Swin Transformer module based on the RBF attention mechanism. X0 passes through each encoding sub-block E i A scale feature will be generated

[0041] c) Feature X4 extracts high-order global features through the bottleneck layer B Then Y4 passes through four decoding sub-blocks (D i , i=4,3,2,1) series decoder, each decoding sub-block contains an upsampling module and a feature conversion module, where the feature conversion module contains two convolution layers with a convolution kernel size of 3×3 and an identity link, and each convolution layer is followed by a LeakyReLU activation layer. Decoding sub-block D i First, the input feature Y i Upsampling, then the upsampled features are compared with the same level coding sub-block E i Input features X i-1 Parallel along the channel dimension, and finally sent to the feature conversion module to generate the output feature Y i-1 After passing through the decoder, high-resolution visual features are generated in turn.

[0042] d) Feature Y0 passes through the output projection layer P out Generate the residual image of the input image I The final output of the image denoising network is obtained through f θ (I) = I + R acquisition.

[0043] Step S3. Construct loss function and train network. Take a batch of image pairs The batch size is N, and I is fed into the network to obtain the output image f θ (I), and then calculate the loss function:

[0044]

[0045] where ε is 1×10 -6The AdamW optimizer is then used to update the network parameters. This step is repeated until the number of training rounds reaches 250, and the parameters with the highest PSNR index on the test set are finally saved.

[0046] Step S4. Test a new image. First, transform the input image's numerical range from [0, 255] to [-1, +1]. Then load the model and trained parameters. Input the image into the network to complete a forward pass. Finally, requantize the network's output image's numerical range to [0, 255] to obtain the denoised image.

[0047] It should be understood that the above description of the implementation examples is relatively specific and cannot be regarded as limiting the scope of patent protection of the present invention. The scope of patent protection of the present invention should be based on the attached claims.

Claims

1. An image denoising method based on an asymmetric structure network, characterized in that: The following steps are involved: S1. Collect images containing real noise and their corresponding noise-free images, preprocess the collected data, and then divide them into training and test sets; S2. Construct an asymmetric image denoising network. The network adopts a codec structure, where the encoder is mainly constructed by a large-capacity Swin Transformer module, and the decoder is mainly composed of computationally efficient residual convolution. The network specifically consists of five parts, in order: input projection layer P in , encoder E, bottleneck layer B, decoder D, output projection layer P out， Where a) input projection layer P in It consists of a convolution layer with a convolution kernel size of 3×3 and a LeakyReLU activation layer; b) The encoder E consists of four consecutive coding sub-blocks {E1, E2, E3, E4}, each coding sub-block E i Contains a feature conversion module and a downsampling module; c) The bottleneck layer B contains a feature conversion module; d) The decoder D consists of four serial decoding sub-blocks {D4, D3, D2, D1}, each decoding sub-block D i Contains an upsampling module and a feature conversion module; e) Output projection layer P out It consists of a convolution layer with a convolution kernel size of 3×3. These five parts are organized together in series, and each coding sub-block E i Decoding sub-block D of the same level i There is a skip link for the transfer of information; S3. Construct a loss function and then use the training set to optimize the image denoising network. After each round of training, calculate the PSNR value of the network on the test set. Finally, save the parameter with the highest PSNR for subsequent inference testing. S4. Test new data by inputting the new image into the trained image denoising network for inference, and then post-processing the network output to obtain the denoised image.

2. The method according to claim 1, wherein: The preprocessing described in step S1 is to crop the collected image pairs, cut the high-resolution image into multiple image slices of 256×256 size at equal intervals, and then convert the sliced ​​image value range from [0, 255] to [-1, +1]. The conversion rules are as follows: Where T in is the image value before conversion, T out is the converted image value.

3. The method according to claim 1, wherein: The feature conversion module described in step S2 b) is a Swin Transformer module using the RBF attention mechanism, which includes a Transformer without an offset window and a Transformer using an offset window, where the size of each window is set to 8×8; secondly, the downsampling module in b) is a convolutional layer with a stride of 2 and a convolution kernel size of 4×4.

4. The method according to claim 1, wherein: The feature conversion module described in step S2 c) is a global Transformer module. The specific operation is to flatten the input features along the spatial dimension, then use the Transformer to model the relationship between the flattened features, and finally convert the output features to the same size as the input features.

5. The method according to claim 1, wherein: The upsampling module described in step S2 d) includes a neighboring interpolation upsampling layer with a sampling factor of 2 and a convolution layer with a convolution kernel size of 3×3; the feature conversion module in d) includes two convolution layers with a convolution kernel size of 3×3 and an identity link, and each convolution layer is followed by a LeakyReLU activation layer.

6. The method according to claim 1, wherein: The loss function described in step S3 is the Charbonnier loss function: Where I is the input image, is the corresponding noise-free image, ||*|| represents the L2 norm, f is the denoising network, θ is the parameter of the denoising network, and ε is 1×10 -6 .

7. The method according to claim 1, wherein: The post-processing described in step S4 is to re-quantize the output image and transform the value range of the output image from [-1, +1] to [0, 255]. The calculation rules are as follows: Where T in Output image value for the network, T out is the post-processed value, and round is the rounding operation.

8. The method according to claim 3, wherein: The calculation process of the RBF attention mechanism is as follows: First, the input feature vector Projection is performed, where n is the length of the feature and d represents the dimension of each feature. After projection, three projection matrices are obtained respectively Where Q = Z in W Q ,K=Z in W K ,V=Z in W V , the expression of RBF attention mechanism is: in is the output feature vector of the RBF attention mechanism. The norm(*) operation performs L2 normalization on the row vector of the input matrix. λ is a hyperparameter with a default value of 0.02.

Images