A deep learning algorithm for image denoising based on blind spot self-supervision
By introducing blind spot self-supervision mechanism and multi-dimensional feature fusion into the image denoising algorithm, the dependence on a large number of paired image data and noise distribution assumptions in the prior art is solved, and an efficient image denoising effect is achieved.
Patent Information
- Application Number
- CN202310722880.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-19
- Publication Date
- 2025-04-29
- Estimated Expiration
- 2043-06-19
AI Technical Summary
Existing image denoising algorithms require a large number of pairs of noisy and clean images for supervised learning, and assuming that the noise distribution is known, Transformer technology is insufficient in self-supervised denoising.
A deep learning algorithm for image denoising based on blind spot self-supervised image denoising is designed. Self-supervised feature extraction is achieved by adding masks to the feature extraction module of Swin-Transformer. Inspired by UNet, different pixels P are selected for encoding and decoding of features, and finally fuse multi-dimensional features to obtain the final denoised image.
It realizes effective image denoising without relying on a large number of paired image data, and performs excellently on open source data sets, solving the problem of insufficient application of noise distribution assumptions and Transformer technology in the prior art.
Smart Images
Figure CN116739932B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image generation, and in particular to an image denoising deep learning algorithm based on blind spot self-supervision.
[0002] Background technology.
[0003] The goal of image denoising is to reconstruct the true value of a given noisy input. The noise depends on aspects such as the camera sensor, signal processor, and image acquisition process, and can take the form of Gaussian noise, Poisson noise, salt and pepper noise, etc. The method proposed by Lehtinen et al. requires a large number of paired noisy and clean images for supervised learning. This method requires the collection of multiple image samples, which may not exist in existing datasets.
[0004] The blind spot feature is usually implemented by masking, which can effectively solve the problem of small receptive field of traditional convolutional neural networks. Honzatko et al. proposed a blind spot convolution and designed an architecture with ten such layers in the decoder to aggregate information from various levels of the encoder. The main limitation of this method is that it assumes that the noise distribution is known, and the task of the network is only to estimate its parameters.
[0005] Transformer techniques are widely used for image restoration in supervised settings, but rarely for self-supervised denoising. Liu et al. constructed a single image denoising transformer using self-attention blocks interleaved with convolutional layers, but the architecture was not tested for self-supervised denoising on a large dataset.
[0006] In summary, the present invention solves the existing problems by designing an image denoising deep learning algorithm based on blind spot self-supervision. Summary of the invention
[0007] In view of the defects of the current image denoising algorithms and the successful application of Transformer technology, the present invention provides an image denoising deep learning algorithm based on blind spot self-supervision. A mask is added to the feature extraction module of Swin-Transformer to realize self-supervised feature extraction. At the same time, inspired by UNet, different pixels P are selected for feature encoding and decoding, and the encoding and decoding inputs of each pixel P are independent. Finally, multi-dimensional features are fused to obtain the final denoised image.
[0008] To achieve the above object, the present invention provides the following technical solutions:
[0009] A deep learning algorithm for image denoising based on blind spot self-supervision includes the following steps:
[0010] Step S1, data construction and data preprocessing: The open source datasets for image denoising algorithms are mainly BSD68Dataset, ImageNet Dataset and sRGB Dataset. If you need to train on your own dataset, you need to prepare the data format to be consistent with the above datasets;
[0011] Step S2, Pixel-ST backbone network construction: The network for image feature extraction is an improved Swin-Transformer, denoted as Pixel-ST. The input image is linearly mapped to obtain the query, key, and value of the image feature. The query, key, and value are shuffled and segmented according to the size of the pixel P, and then the query is normalized. Next, the diagonal of the input attention matrix is masked so that each pixel does not pay attention to its own value. In the next stage of attention matrix calculation, each pixel cannot access the state of the corresponding pixel in the previous level to achieve the purpose of self-supervision; after the mask feature is extracted, the query, key, and value are spliced, and the original value of the query is cross-connected and fused. The regularization of the last layer and the fusion of the multi-layer perceptron obtain the Pixel-ST feature extraction result;
[0012] Step S3, Pixel-UNet network overall architecture: The overall network architecture design follows the UNet network structure, which is a global skip connection idea. The pixel P (P=1, 2, 4, 6) is set, and the network design consists of four Pixel-ST feature encoders and three Pixel-ST decoders. The pixel 1 is operated on the pixel-level image, and the other pixels P are down-sampled operations on the image. In order to meet the blind spot characteristics, the key and value input sizes of the feature extraction blocks of different pixel sizes are fixed input data values;
[0013] Step S4, preprocessing the input image, generating a noisy image using Gaussian noise or Poisson noise, inputting it into the network architecture of the UNet paradigm, and obtaining the query, key, and value of the image features through linear mapping. After mask self-supervised feature extraction and blind spot multi-dimensional feature fusion, the final denoised image is obtained.
[0014] As a preferred solution of the present invention, the image in S1 has a length of 512, a width of 512, and a channel number of 3.
[0015] As a preferred solution of the present invention, in S2, a noisy image is generated using Gaussian noise or Poisson noise as input data. First, the query, key, and value parts of the image feature are obtained through linear mapping according to the size of the input image. Here, the query is the position encoding of all windows and attention heads. After obtaining the query, key, and value, the query is normalized. Secondly, the normalized query and key and value are input into the self-supervision module, and the diagonal of the attention matrix is masked. In addition, the connection and linear projection of the query are skipped around each block to the original embedding dimension, thereby obtaining a new query. At this point, the improved Swin-Transformer feature extraction block is completed, denoted as Pixel-ST.
[0016] As a preferred solution of the present invention, the input of the overall network architecture in S3 is an image with Gaussian noise or Poisson noise. After obtaining the query, key, and value in step S2, the query is propagated through the blocks in sequence, while the key and value are fixed and embedded and summed in the same position as the query, and passed unchanged to each Pixel-ST feature extraction block to maintain input isolation; the query, key, and value are respectively input into the image encoding of pixel P (P=1, 2, 4, 6), and according to the structure of UNet, the query output by the Pixel-ST feature extraction block of pixel P=6 is concat-operated with the query output by the Pixel-ST feature extraction block of pixel P=4, and the result is used as the output of the decoder of pixel P=4, and the same operation is performed on pixels P=2 and 1.
[0017] As a preferred solution of the present invention, in S4, the image is preprocessed, and a noisy image is generated using Gaussian noise or Poisson noise, which is input into the network architecture of the UNet paradigm. After the features of multiple dimensions of pixels P (P=1, 2, 4, 6) are fused, the final denoised image is obtained.
[0018] As a preferred solution of the present invention, the query is Q, the key is K, and the value is V.
[0019] Compared with the prior art, the present invention has the following beneficial effects:
[0020] In view of the defects of the current image denoising algorithms and the successful application of Transformer technology, the present invention provides an image denoising deep learning algorithm based on blind spot self-supervision. A mask is added to the feature extraction module of Swin-Transformer to realize self-supervised feature extraction. At the same time, inspired by UNet, different pixels P are selected for feature encoding and decoding, and the encoding and decoding input of each pixel P is independent. Finally, multi-dimensional features are integrated to obtain the final denoised image, which ranks at the forefront in open source datasets. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 This is the Piexl-ST backbone network structure diagram of the present invention;
[0022] Figure 2 This is the Pixel-UNet network architecture diagram of the present invention. Implementation
[0023] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0024] To facilitate understanding of the present invention, the present invention will be described more comprehensively below with reference to the relevant drawings, and several embodiments of the present invention are given. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.
[0025] It should be noted that when an element is referred to as being "fixed on" another element, it may be directly on the other element or there may also be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used in this article are for illustrative purposes only.
[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which the present invention belongs. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used in this article includes any and all combinations of one or more related listed items.
[0027] For examples, see Figure 1-2 , the present invention provides a technical solution:
[0028] The invention provides an image denoising deep learning algorithm based on blind spot self-supervision. The algorithm only uses a camera image as an input. First, a noisy image is generated by using Gaussian noise or Poisson noise. After linear mapping, a query (Q), a key (K), and a value (V) of an image feature are obtained, which are input into a network architecture of a UNet paradigm. Feature encoding is first performed, and Q, K, and V are input into a Pixel-ST feature extraction block with a pixel of 1. After shuffling and segmentation, the query (Q) is normalized, self-supervised mask feature extraction is performed, Q, K, and V are spliced, and the original value of the query (Q) is cross-connected and fused. After regularization and fusion of a multi-layer perceptron, a cross-connected fusion operation is finally performed with the original value to obtain a Pixel-ST feature extraction result with a pixel of 1. Then, Q, K, and V are input into a Pixel-ST feature extraction block with pixels of 2, 4, and 6, and the same steps as the pixel of 1 are performed to obtain feature encoding results of pixels of 2, 4, and 6 in sequence. Next, a decoding operation is performed, except for the pixel of 6. , the remaining pixels are encoded and decoded in pairs, the Pixel-ST feature encoding result with 6 pixels is cross-connected and fused with the original value, and then fused (Concat) with the Pixel-ST feature encoding result with 4 pixels as the input of the Pixel-ST feature decoding with 4 pixels, the result of the Pixel-ST feature decoding with 4 pixels is cross-connected and fused with the original value as the input of the Pixel-ST feature decoding with 2 pixels, the result of the Pixel-ST feature decoding with 2 pixels is cross-connected and fused with the original value as the input of the Pixel-ST feature decoding with 1 pixel, and the result of the Pixel-ST feature decoding with 1 pixel is the final denoised image. The present invention mainly includes the following steps: data construction, data preprocessing, Pixel-ST backbone network construction, and Pixel-UNet network design.
[0029] The following is an explanation of the adopted technical solutions:
[0030] Data construction, data preprocessing, Pixel-ST backbone network construction, Pixel-UNet network overall architecture
[0031] Step S1, data construction and data preprocessing: The open source datasets for image denoising algorithms are mainly BSD68Dataset, ImageNet Dataset and sRGB Dataset. If you need to train on your own dataset, you need to prepare the data format to be consistent with the above datasets;
[0032] Step S2, Pixel-ST backbone network construction: The network for image feature extraction is an improved Swin-Transformer, denoted as Pixel-ST. The input image is linearly mapped to obtain the query (Q), key (K), and value (V) of the image features. Q, K, and V are shuffled and segmented according to the size of pixel P, and the query (Q) is normalized. Next, the diagonal of the input attention matrix is masked so that each pixel does not pay attention to its own value. In the next stage of attention matrix calculation, each pixel cannot access the state of the corresponding pixel in the previous level to achieve the purpose of self-supervision. After the mask feature extraction, Q, K, and V are spliced, and the original value of the query (Q) is cross-connected and fused. The regularization of the last layer and the fusion of the multi-layer perceptron obtain the Pixel-ST feature extraction result. See the specific network result diagram. Figure 1 ;
[0033] Step S3, the overall architecture of the Pixel-UNet network: input Q, K, and V into the Pixel-ST feature extraction block with a pixel of 1, and perform shuffle segmentation, normalization of the query (Q), self-supervised mask feature extraction, concatenation of the query (Q), key (K), and value (V), and cross-connection fusion of the original value of the query (Q). After regularization and fusion of the multi-layer perceptron, the cross-connection fusion operation is finally performed with the original value to obtain the Pixel-ST feature extraction result with a pixel of 1; then input the query (Q), key (K), and value (V) into the Pixel-ST feature extraction block with pixels of 2, 4, and 6, and perform the same steps as for pixel 1 to obtain the feature encoding results of pixels 2, 4, and 6 in turn, and then perform a decoding operation, except for the pixel 6. , the remaining pixels are encoded and decoded in pairs, the Pixel-ST feature encoding result of pixel 6 is cross-connected and fused with the original value, and then fused (Concat) with the Pixel-ST feature encoding result of pixel 4 as the input of Pixel-ST feature decoding of pixel 4, the result of Pixel-ST feature decoding of pixel 4 is cross-connected and fused with the original value as the input of Pixel-ST feature decoding of pixel 2, the result of Pixel-ST feature decoding of pixel 2 is cross-connected and fused with the original value as the input of Pixel-ST feature decoding of pixel 1, and the Pixel-ST feature decoding of pixel 1 is the final denoised image. The overall Pixel-UNet network architecture is as follows Figure 2 .
[0034] The specific implementation of the monocular camera depth estimation algorithm based on piecewise plane prior is divided into the following parts:
[0035] Step 1: Data preparation
[0036] The open source datasets for image denoising algorithms are mainly BSD68 Dataset, ImageNet Dataset, and sRGBDataset. If you need to train on your own dataset, you need to prepare the data format to be consistent with the above datasets;
[0037] Step 2: Training phase
[0038] The input is a 512*512*3 camera image. First, a noisy image is generated using Gaussian noise or Poisson noise. After linear mapping, the query (Q), key (K), and value (V) of the image features are obtained and input into the network architecture of the UNet paradigm. First, feature encoding is performed. The query (Q), key (K), and value (V) are input into the Pixel-ST feature extraction block with a pixel of 1. After shuffling and segmentation, the query (Q) is normalized, self-supervised mask feature extraction is performed, and the query (Q), key (K), and value (V) are concatenated. The original value of the query (Q) is cross-connected and fused. After regularization and fusion of multi-layer perceptrons, the cross-connection fusion operation is finally performed with the original value to obtain the Pixel-ST feature extraction result with a pixel of 1. Then Q, K, and V are input into the Pixel-ST feature extraction block with pixels of 2, 4, and 6. The same steps as for pixel 1 are performed to obtain the feature encoding results of pixels 2, 4, and 6 in turn. Next, a decoding operation is performed. Except for the pixel 6 , the remaining pixels are encoded and decoded in pairs. The encoding result of the Pixel-ST feature with a pixel of 6 is cross-connected and fused with the original value, and then fused (Concat) with the Pixel-ST feature encoding result with a pixel of 4 as the input of the Pixel-ST feature decoding with a pixel of 4. The result of the Pixel-ST feature decoding with a pixel of 4 is cross-connected and fused with the original value as the input of the Pixel-ST feature decoding with a pixel of 2. The result of the Pixel-ST feature decoding with a pixel of 2 is cross-connected and fused with the original value as the input of the Pixel-ST feature decoding with a pixel of 1. Finally, the result of the Pixel-ST feature decoding with a pixel of 1 is the final denoised image.
[0039] Step 3: Testing phase
[0040] Input the test image and send it to the Pixel-UNet network for inference to obtain the final denoised image and calculate the final accuracy of the peak signal-to-noise ratio (PSNR) measurement algorithm.
[0041] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A deep learning algorithm for image denoising based on blind spot self-supervision, comprising the following steps: Step S1, data construction and data preprocessing: The open source datasets for image denoising algorithms are mainly BSD68 Dataset, ImageNet Dataset and sRGB Dataset. If you need to train on your own dataset, you need to prepare the data format to be consistent with the above datasets; Step S2, Pixel-ST backbone network construction: The network for image feature extraction is an improved Swin-Transformer, denoted as Pixel-ST. The input image is linearly mapped to obtain the query, key, and value of the image feature. The query, key, and value are shuffled and segmented according to the size of the pixel P, and then the query is normalized. Next, the diagonal of the input attention matrix is masked so that each pixel does not pay attention to its own value. In the next stage of attention matrix calculation, each pixel cannot access the state of the corresponding pixel in the previous level to achieve the purpose of self-supervision; After mask feature extraction, query, key, and value are concatenated, and the original query value is fused across connections. The regularization of the last layer and the fusion of the multi-layer perceptron obtain the Pixel-ST feature extraction result. Step S3, Pixel-UNet network overall architecture: The overall network architecture design follows the UNet network structure, which is a global skip connection idea. The pixel P (P=1, 2, 4, 6) is set, and the network design consists of four Pixel-ST feature encoders and three Pixel-ST decoders. The pixel 1 is operated on the pixel-level image, and the other pixels P are down-sampled operations on the image. In order to meet the blind spot characteristics, the key and value input sizes of the feature extraction blocks of different pixel sizes are fixed input data values; Step S4, preprocessing the input image, generating a noisy image using Gaussian noise or Poisson noise, inputting it into the network architecture of the UNet paradigm, and obtaining the query, key, and value of the image features through linear mapping. After mask self-supervised feature extraction and blind spot multi-dimensional feature fusion, the final denoised image is obtained.
2. The image denoising deep learning algorithm based on blind spot self-supervision according to claim 1, characterized in that: The image in S1 has a length of 512, a width of 512, and a channel number of 3.
3. The image denoising deep learning algorithm based on blind spot self-supervision according to claim 1, characterized in that: In S2, a noisy image is generated using Gaussian noise or Poisson noise as input data. First, a linear mapping is performed according to the size of the input image to obtain the query, key, and value parts of the image feature. Here, the query is the position encoding of all windows and attention heads. After obtaining the query, key, and value, the query is normalized. Secondly, the normalized query and key and value are input into the self-supervision module, and the diagonal of the attention matrix is masked. In addition, the connection and linear projection of the query are skipped around each block to the original embedding dimension, thereby obtaining a new query. At this point, the improved Swin-Transformer feature extraction block is completed, denoted as Pixel-ST.
4. The image denoising deep learning algorithm based on blind spot self-supervision according to claim 1, characterized in that: The input of the overall network architecture in S3 is an image with Gaussian noise or Poisson noise. After obtaining the query, key, and value in step S2, the query is propagated through the blocks in sequence, while the key and value are fixed and embedded and summed with the same position as the query, and passed unchanged to each Pixel-ST feature extraction block to keep the input isolated; The query, key, and value are input into the image encoding of pixel P (P=1, 2, 4, 6) respectively. According to the structure of UNet, the query output by the Pixel-ST feature extraction block of pixel P=6 is concat-operated with the query output by the Pixel-ST feature extraction block of pixel P=4, and the result is used as the output of the decoder of pixel P=4. Similarly, the same operation is performed on pixels P=2 and 1.
5. The image denoising deep learning algorithm based on blind spot self-supervision according to claim 1, characterized in that: In the S4, the image is preprocessed, and a noisy image is generated using Gaussian noise or Poisson noise, which is input into the network architecture of the UNet paradigm. After the features of multiple dimensions of pixels P (P=1, 2, 4, 6) are fused, the final denoised image is obtained.
6. The image denoising deep learning algorithm based on blind spot self-supervision according to claim 1, characterized in that: The query is Q, the key is K, and the value is V.