Raw domain image denoising network based on prior guidance

Abstract

The present application relates to RAW domain image denoising, in particular to a kind of RAW domain image denoising network based on priori guide, adopt UNet network as main network architecture, including encoder and decoder, including first convolution layer in the encoder, and multiple encoder units consisting of priori guide's conversion module, down-sampling layer, first convolution layer is cascaded multiple encoder units after stage;Multiple decoder units consisting of up-sampling layer, priori guide's conversion module and second convolution layer are included in the decoder, multiple decoder units are connected after cascading second convolution layer;Priori guide's conversion module includes cascaded normalization layer, priori guide's luminance attention module PGLA, priori guide's feedforward neural network PGFFN;The technical scheme provided by the present application can effectively overcome the defects that the RAW domain image denoising effect is poor and the RAW domain image denoising generalization ability is insufficient in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to RAW domain image denoising, in particular to a RAW domain image denoising network based on prior guidance. BACKGROUND

[0002] Image denoising, as a basic research direction in the field of computer vision, mainly processes two kinds of image formats: RAW images and RGB images. In recent years, the academic community has proposed a large number of efficient RAW image and RGB image denoising algorithms, and has made significant research achievements. Compared with RGB images, RAW format has higher bit depth and can preserve more rich original image details. At the same time, since RAW images are not processed by image signal processors (ISPs) in the later stage, their noise distribution is usually simpler and more regular than that of RGB images. Based on these characteristics, direct denoising processing in the RAW domain can often more effectively restore image details and texture information.

[0003] Existing RAW domain image denoising methods, such as Physics-based Noise Modeling for Extreme Low-light Photography proposed by Kaixuan Wei et al. in 2021 and TSDN: Two-Stage Raw Denoising in the Dark proposed by Wenshu Chen et al. in 2023, these data-driven methods, through automatic feature extraction, end-to-end learning and powerful nonlinear modeling capability, have made certain breakthroughs in RAW domain image denoising. However, these methods only learn the paired relationship between training data, and the model is heavily dependent on the distribution of training data, therefore, the existing methods are difficult to effectively apply to the RAW domain image denoising task in real-world scenarios. SUMMARY

[0004] (I) Technical problems solved

[0005] In view of the above shortcomings of the prior art, the present application provides a RAW domain image denoising network based on prior guidance, which can effectively overcome the defects of poor RAW domain image denoising effect and insufficient generalization ability of RAW domain image denoising of the prior art.

[0006] (II) Technical solutions

[0007] To achieve the above purpose, the present application is realized by the following technical solutions:

[0008] The priori-guided RAW domain image denoising network adopts a UNet network as a backbone network architecture, and includes an encoder and a decoder, the encoder includes a first convolutional layer and a plurality of encoder units composed of a priori-guided conversion module and a down-sampling layer, and the first convolutional layer is connected with the plurality of encoder units in cascade;

[0009] The decoder includes a plurality of decoder units composed of an up-sampling layer, a priori-guided conversion module and a second convolutional layer, and the plurality of decoder units are connected with the second convolutional layer in cascade;

[0010] The priori-guided conversion module includes a normalization layer, a priori-guided luminance attention module PGLA and a priori-guided feedforward neural network PGFFN in cascade, the PGLA inputs the light component and the reflection component of the RAW domain image to guide the model to better learn the noise region in the RAW domain image, and the PGFFN inputs the camera sensor data priori information to guide the model to better adapt to different noise types and noise intensities in the RAW domain image, and enhance the generalization ability of the model to different noises.

[0011] Preferably, the light component and the reflection component of the RAW domain image are estimated by a light prior estimation module, and for any RAW domain image , the light prior estimation module first calculates the mean value along the channel dimension, then splices the mean value with the RAW domain image X, and after passing through a 3*3 convolutional layer, respectively passes through two 5*5 deep convolutional layers and a GELU activation function to estimate the light component and the reflection component of the RAW domain image X.

[0012] Wherein, C in is the channel number of the RAW domain image X, C is the channel number of the processed RAW domain image X, H is the height of the RAW domain image X, and W is the width of the RAW domain image X.

[0013] Preferably, in the encoder, after the RAW domain image X passes through a 3*3 first convolutional layer, it enters the multi-stage processing process of the encoder unit:

[0014] ;

[0015] Wherein, X i , Y irespectively, and L, R are light component and reflectance component of the prior guided luminance attention module PGLA in the prior guided conversion module of the i-th encoder unit, M is the camera sensor data prior information of the prior guided feedforward neural network PGFFN in the prior guided conversion module of the i-th encoder unit, and the camera sensor data prior information extracted from the RAW domain image X, N is the number of camera sensor data prior information, F represents the prior guided conversion module, and down represents a down-sampling layer including convolution and pixel inverse shuffling processing.

[0016] Preferably, in the prior guided conversion module of the i-th encoder unit, the light component L and the reflectance component R of the RAW domain image X are input into the prior guided luminance attention module PGLA, and the camera sensor data prior information M is input into the prior guided feedforward neural network PGFFN.

[0017] ;

[0018] wherein Y i is the intermediate output of the prior guided conversion module of the i-th encoder unit, and ln represents a normalization layer.

[0019] Preferably, in the prior guided luminance attention module PGLA of the prior guided conversion module of the i-th encoder unit, the RAW domain image X is first processed by two convolution layers to obtain feature values V1:

[0020] ;

[0021] wherein conv1 represents a 1*1 convolution layer, and conv3 represents a 3*3 convolution layer.

[0022] Meanwhile, the RAW domain image X is reshaped and processed by three different linear transformations to obtain a query matrix Q, a key matrix K and a value matrix V2:

[0023] ;

[0024] wherein Reshape represents a reshaping operation, represents a linear transformation.

[0025] The attention score matrix A is calculated using the query matrix Q and the key matrix K, and then the attention score matrix A is copied multiple times to obtain an extended attention score matrix A':

[0026] ;

[0027] wherein softmax represents a softmax function.

[0028] Meanwhile, the light component L and the reflection component R of the RAW domain image X are reshaped and linearly transformed in two different ways, and then fused with the value matrix V2 through element-wise multiplication and addition, and the fused value matrix V2' is spliced with the eigenvalue V1, and then linearly transformed and reshaped to obtain the output value matrix V2'':

[0029] ;

[0030] Wherein, L', R' are the processed light component and reflection component respectively, and cat represents the splicing operation;

[0031] The output value matrix V2'' is cross-multiplied with the extended attention score matrix A', and then linearly transformed, reshaped and a convolution layer to obtain the output of the prior guided luminance attention module PGLA in the prior guided conversion module in the i-th encoder unit :

[0032] .

[0033] Preferably, in the prior guided feedforward neural network PGFFN of the prior guided conversion module of the i-th encoder unit, the RAW domain image X and the camera sensor data prior information M are input into the prior fusion sub-module, and the RAW domain image X and the camera sensor data prior information M are first fused through splicing and element-wise multiplication:

[0034] ;

[0035] Wherein, M1, M2 are the first intermediate camera sensor data prior information and the second intermediate camera sensor data prior information respectively, , which means that the camera sensor data prior information M is expanded from C*1 to C*H*W, and X1', X2' are the first fused image and the second fused image respectively;

[0036] The output of the prior fusion sub-module is obtained by using several convolution operations and residual connections :

[0037] ;

[0038] Wherein, Dwconv3 represents a 3*3 deep convolution layer;

[0039] The output of the prior fusion sub-module is input into the prior guided feedforward neural network PGFFN, which is composed of convolution layers and gating operations, to obtain the output of the prior guided feedforward neural network PGFFN of the prior guided conversion module of the i-th encoder unit :

[0040] ;

[0041] wherein X1'', X2'' are the first dimension split image, the second dimension split image respectively, chunk represents a dimension split operation on the target, and the target is divided into two parts with the same dimension, that is, the dimensions of the first dimension split image X1'' and the second dimension split image X2'' are , and GELU represents a GELU activation function.

[0042] (Three) beneficial effects

[0043] Compared with the prior art, the RAW domain image denoising network based on prior guidance provided by the present application has the following beneficial effects:

[0044] 1) The RAW domain image denoising network based on prior guidance proposed by the present application is the first RAW domain image denoising network combining illumination prior and camera sensor prior, and can effectively remove the noise of the RAW domain image;

[0045] 2) The prior guided brightness attention module PGLA proposed by the present application better learns the noise area in the RAW domain image through the illumination intensity, and improves the denoising effect of the model;

[0046] 3) The prior guided feedforward neural network PGFFN proposed by the present application realizes effective perception of different noise types and noise intensities through the camera sensor data prior information, and enhances the generalization ability of the model to different scenes and noises;

[0047] 4) The RAW domain image denoising network based on prior guidance proposed by the present application has only a parameter amount of 6.179MB, and has a real-time inference speed of 0.25s, and can be deployed on edge devices and realize real-time removal of RAW domain image noise. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0049] Figure 1 is the training flowchart of the RAW domain image denoising network based on prior guidance in the present application;

[0050] Figure 2 is a noisy RAW domain image;

[0051] Figure 3 To utilize the prior-guided RAW domain image denoising network proposed in this invention... Figure 2 The RAW domain denoised image obtained by denoising. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0053] The following describes the specific network architecture of the RAW domain image denoising network based on prior guidance provided by this invention, using concrete examples. The system network architecture includes:

[0054] The UNet network is used as the backbone network architecture, which includes an encoder and a decoder. The encoder includes a first convolutional layer and multiple encoder units consisting of prior-guided transformation modules and downsampling layers. Multiple encoder units are cascaded after the first convolutional layer.

[0055] The decoder includes multiple decoder units consisting of upsampling layers and prior-guided conversion modules, as well as a second convolutional layer. Multiple decoder units are cascaded and then connected to the second convolutional layer.

[0056] The prior-guided transformation module includes a cascaded normalization layer, a prior-guided luminance attention module PGLA, and a prior-guided feedforward neural network PGFFN. The prior-guided luminance attention module PGLA is input with the light and reflection components of the RAW domain image to guide the model to better learn the noisy regions in the RAW domain image. The prior-guided feedforward neural network PGFFN is input with prior information from the camera sensor data to guide the model to better adapt to different noise types and intensities in the RAW domain image, thereby enhancing the model's generalization ability to different noises.

[0057] I. Optical Prior Estimation Module

[0058] The optical and reflection components of the RAW domain image are estimated through an optical prior estimation module. For any RAW domain image... The optical prior estimation module first calculates the mean along the channel dimension, then concatenates this mean with the RAW domain image X. After passing through a 3*3 convolutional layer, it then passes through two 5*5 depth convolutional layers and the GELU activation function to estimate the optical components of the RAW domain image X. and reflection component ;

[0059] wherein C in is the number of channels of the RAW domain image X, C is the number of channels of the processed RAW domain image X, H is the height of the RAW domain image X, and W is the width of the RAW domain image X.

[0060] II. Encoder

[0061] In the encoder, the RAW domain image X is processed by a first convolutional layer of 3*3 and then enters the multi-stage processing procedure of the encoder unit:

[0062] ;

[0063] wherein X i , Y i are the input and output of the prior-guided conversion module in the i-th encoder unit, L and R are the light component and the reflection component of the prior-guided luminance attention module PGLA in the prior-guided conversion module input into the i-th encoder unit, and M is the camera sensor data prior information of the prior-guided feedforward neural network PGFFN in the prior-guided conversion module input into the i-th encoder unit, the camera sensor data prior information is extracted from the RAW domain image X, N is the number of the camera sensor data prior information, F represents the prior-guided conversion module, and down represents a down-sampling layer including convolution and pixel inverse shuffling processing.

[0064] In the technical solution of the present application, the working principle of the decoder is similar to that of the above-mentioned encoder.

[0065] III. Prior-guided conversion module of the encoder unit

[0066] In the prior-guided conversion module of the i-th encoder unit, the light component L and the reflection component R of the RAW domain image X are input into the prior-guided luminance attention module PGLA, and the camera sensor data prior information M is input into the prior-guided feedforward neural network PGFFN:

[0067] ;

[0068] wherein Y i is the intermediate output of the prior-guided conversion module in the i-th encoder unit, and ln represents a normalization layer.

[0069] In the technical solution of the present application, the working principle of the prior-guided conversion module of the decoder unit is similar to that of the above-mentioned prior-guided conversion module of the encoder unit.

[0070] IV. Prior-guided luminance attention module PGLA of the prior-guided conversion module of the encoder unit

[0071] In the priori-guided luminance attention module PGLA of the priori-guided conversion module of the ith encoder unit, the RAW domain image X is first subjected to two convolution layers to obtain feature values V1:

[0072] ;

[0073] wherein conv1 represents a 1*1 convolution layer, and conv3 represents a 3*3 convolution layer;

[0074] Meanwhile, the RAW domain image X is reshaped and subjected to three different linear transformations to obtain a query matrix Q, a key matrix K and a value matrix V2:

[0075] ;

[0076] wherein Reshape represents a reshaping operation, and Linear represents a linear transformation;

[0077] The attention score matrix A is calculated using the query matrix Q and the key matrix K, and then the attention score matrix A is copied multiple times to obtain an extended attention score matrix A':

[0078] ;

[0079] wherein softmax represents a softmax function;

[0080] Meanwhile, the light component L and the reflection component R of the RAW domain image X are subjected to reshaping and two different linear transformations, and then fused with the value matrix V2 through element-wise multiplication and addition. The fused value matrix V2' is spliced with the feature values V1, and then subjected to linear transformation and reshaping to obtain an output value matrix V2'':

[0081] ;

[0082] wherein L' and R' are the processed light component and reflection component respectively, and cat represents a splicing operation;

[0083] The output value matrix V2'' is subjected to a cross product operation with the extended attention score matrix A', and then subjected to linear transformation, reshaping and a convolution layer to obtain the output of the priori-guided luminance attention module PGLA in the priori-guided conversion module in the ith encoder unit :

[0084] .

[0085] In the technical solution of the present application, the prior-guided luminance attention module PGLA of the prior-guided conversion module of the decoder unit has a working principle similar to that of the prior-guided luminance attention module PGLA of the prior-guided conversion module of the encoder unit.

[0086] V. Prior-guided feedforward neural network PGFFN of the prior-guided conversion module of the encoder unit

[0087] In the prior-guided feedforward neural network PGFFN of the prior-guided conversion module of the i-th encoder unit, the RAW domain image X and the camera sensor data prior information M are input into the prior fusion sub-module, and the RAW domain image X and the camera sensor data prior information M are first fused by splicing and element-wise multiplication:

[0088] ;

[0089] wherein M1 and M2 are the first intermediate camera sensor data prior information and the second intermediate camera sensor data prior information, respectively, indicates that the camera sensor data prior information M is expanded from Cx1 to CxHxW, and X1' and X2' are the first fused image and the second fused image, respectively;

[0090] The prior fusion sub-module uses several convolution operations and residual connections as the output :

[0091] ;

[0092] wherein Dwconv3 represents a 3*3 depth convolution layer;

[0093] The output of the prior fusion sub-module is input into the prior-guided feedforward neural network PGFFN, which is composed of convolution layers and gating operations, to obtain the output of the prior-guided feedforward neural network PGFFN of the prior-guided conversion module of the i-th encoder unit :

[0094] ;

[0095] wherein X1'' and X2'' are the first dimensionally uniform image and the second dimensionally uniform image, respectively, and chunk represents a dimensionally uniform operation on the target, which divides the target into two parts of the same dimension, i.e., the first dimensionally uniform image X1'' and the second dimensionally uniform image X2'' have the same dimension , and GELU represents a GELU activation function.

[0096] In the technical solution of this application, the prior-guided feedforward neural network PGFFN of the prior-guided conversion module of the decoder unit works in a similar principle to the prior-guided feedforward neural network PGFFN of the prior-guided conversion module of the encoder unit.

[0097] Figure 1 The flowchart below shows the training process of the prior-guided RAW domain image denoising network in this invention. First, a UNet network is constructed as the backbone network architecture (including encoder and decoder), and the network parameters are initialized using a uniform distribution initialization method. Then, a training dataset is constructed. The SID dataset contains 409 pairs of noisy original RAW domain images and noise-free RAW domain images. 280 pairs are used as the training set, and the remaining 129 pairs are used as the test set. For each image, the exiftool and the light prior estimation module are used to extract the camera sensor data prior information of the RAW domain image (including exposure time, ISO, aperture value, focal length, brightness value, blue balance, red balance, light value, etc.), as well as the light component and reflection component.

[0098] During training, the loss function is calculated based on the model's output and the corresponding noise-free RAW domain image. The network parameters are then updated using the backpropagation algorithm, and training stops after a preset number of iterations. In practical applications, noisy RAW domain images acquired in real time are input into the pre-trained prior-guided RAW domain image denoising network to directly obtain the denoised RAW domain image.

[0099] Figure 2 For noisy RAW domain images, Figure 3 To utilize the prior-guided RAW domain image denoising network proposed in this invention... Figure 2 The RAW domain denoised image obtained after denoising. As can be seen from the comparison in the figure, the prior-guided RAW domain image denoising network proposed in this invention can effectively remove noise from RAW domain images.

[0100] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A priori-guided RAW domain image denoising network, characterized in that: The UNet network is used as the backbone network architecture, including an encoder and a decoder. The encoder includes a first convolutional layer and multiple encoder units consisting of prior-guided transformation modules and downsampling layers. Multiple encoder units are cascaded after the first convolutional layer. The decoder includes multiple decoder units consisting of upsampling layers and prior-guided conversion modules, as well as a second convolutional layer. The multiple decoder units are cascaded and then connected to the second convolutional layer. The prior-guided conversion module includes a cascaded normalization layer, a prior-guided luminance attention module PGLA, and a prior-guided feedforward neural network PGFFN. The prior-guided luminance attention module PGLA is input with the light component and reflection component of the RAW domain image to guide the model to better learn the noise region in the RAW domain image. The prior-guided feedforward neural network PGFFN is input with prior information from camera sensor data to guide the model to better adapt to different noise types and intensities in the RAW domain image and enhance the model's generalization ability to different noises. In the i In the prior-guided transformation module of each encoder unit, the prior-guided luminance attention module (PGLA) inputs a RAW domain image. X light component L and reflection component R Prior information from camera sensor data is input into the prior-guided feedforward neural network PGFFN. M : ； in, Y i ' For the first i The intermediate output of the priori-guided conversion module in each encoder unit Y i For the first i The output of the priori-guided conversion module in each encoder unit ln This indicates the normalization layer.

2. The RAW domain image denoising network based on prior guidance according to claim 1, characterized in that: The optical and reflection components of the RAW domain image are estimated through an optical prior estimation module. For any RAW domain image... The optical prior estimation module first calculates the mean along the channel dimension, then concatenates this mean with the RAW domain image X. After passing through a 3*3 convolutional layer, it then passes through two 5*5 depth convolutional layers and the GELU activation function to estimate the optical components of the RAW domain image X. and reflection component ; Among them, C in Let X be the number of channels in the RAW domain image, C be the number of channels in the processed RAW domain image, H be the height of the RAW domain image, and W be the width of the RAW domain image.

3. The RAW domain image denoising network based on prior guidance according to claim 2, characterized in that: In the encoder, the RAW domain image X passes through a 3x3 first convolutional layer and then enters the multi-stage processing of the encoder unit: ； Among them, X i Y i Let L and R be the input and output of the prior-guided conversion module in the i-th encoder unit, respectively. Let L and R be the light component and reflection component of the prior-guided luminance attention module PGLA in the prior-guided conversion module of the i-th encoder unit, respectively. Let M be the prior information of the camera sensor data from the prior-guided feedforward neural network PGFFN in the prior-guided conversion module of the i-th encoder unit. Extracted from the RAW domain image X, where N is the amount of prior information from the camera sensor data, F represents the prior-guided transformation module, and down represents the downsampling layer including convolution and pixel inverse shuffling processing.

4. The RAW domain image denoising network based on prior guidance according to claim 1, characterized in that: In the prior-guided luminance attention module PGLA of the prior-guided transformation module of the i-th encoder unit, the RAW domain image X first passes through two convolutional layers to obtain the feature value V1: ； Where conv1 represents a 1*1 convolutional layer and conv3 represents a 3*3 convolutional layer; Simultaneously, the RAW domain image X is reshaped and undergoes three different linear transformations to obtain the query matrix Q, the key matrix K, and the value matrix V2: ； Here, Reshape represents the reshaping operation. Represents a linear transformation; The attention score matrix A is calculated using the query matrix Q and the key matrix K. Then, the attention score matrix A is copied multiple times to obtain the extended attention score matrix A'. ； Where softmax represents the softmax function; Simultaneously, the light component L and reflection component R of the RAW domain image X are reshaped and subjected to two different linear transformations. Then, they are fused with the value matrix V2 through element-wise multiplication and addition. The fused value matrix V2' is then concatenated with the eigenvalues ​​V1, and further linear transformations and reshaping are performed to obtain the output value matrix V2''. ； Where L' and R' are the processed light component and the reflection component, respectively, and cat represents the splicing operation; The output value matrix V2'' is cross-productted with the extended attention score matrix A', followed by linear transformation, reshaping, and a convolutional layer to obtain the output of the prior-guided luminance attention module PGLA in the prior-guided transformation module of the i-th encoder unit. : 。 5. The RAW domain image denoising network based on prior guidance according to claim 4, characterized in that: In the prior-guided feedforward neural network PGFFN of the prior-guided transformation module of the i-th encoder unit, the RAW domain image X and the prior information M of the camera sensor data are input into the prior fusion submodule. The RAW domain image X and the prior information M of the camera sensor data are first fused by concatenation and element-wise multiplication. ； Wherein, M1 and M2 are the prior information of the first intermediate camera sensor data and the prior information of the second intermediate camera sensor data, respectively. This means that the prior information M of the camera sensor data is expanded from C×1 to C×H×W, and X1' and X2' are the first fused image and the second fused image, respectively; The output of the prior fusion submodule is used by several convolution operations and residual connections. : ； Where Dwconv3 represents a 3x3 depth convolutional layer; The output of the prior fusion submodule In the prior-guided feedforward neural network PGFFN, the neural network consists of convolutional layers and gating operations, and obtains the output of the prior-guided feedforward neural network PGFFN of the i-th encoder unit's conversion module. : ； Where X1'' and X2'' are the first-dimensional and second-dimensional equally divided images, respectively, and chunk represents the dimensional division operation performed on the target, dividing the target into two parts with the same dimension, that is, the first-dimensional equally divided image X1'' and the second-dimensional equally divided image X2'' both have the same dimension. GELU represents the GELU activation function.

Images