A black and white image guided joint denoising and demosaicing method for color RAW images

Abstract

This paper discloses a method for joint denoising and demosaicing of color RAW images guided by black and white images. The method comprises the following steps: constructing a black and white-color binocular camera simulated image dataset for training and testing a network module; leveraging the structural correlation between black and white images and color images to construct an alignment guidance image generation module based on a parallax attention mechanism. This module is trained using the true value of the clean grayscale image corresponding to the RAW image as supervision using a perceptual loss function to generate a high-quality alignment guidance image; using the generated alignment guidance image to guide the joint denoising and demosaicing process of the color camera RAW image; and training the guidance denoising and demosaicing module based on a structure-color loss function to ensure the color accuracy of the denoised and demosaiced results while accurately transferring the guidance image structure. This method has strong application value for binocular imaging devices in low-light scenes.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method for joint denoising and demosaicing of color RAW images guided by black and white images. Background Art

[0002] Image denoising and demosaicing are essential parts of the image processing flow of color cameras. Existing methods generally denoise the image in the RAW domain first, and then use the demosaicing algorithm to convert the image into the RGB domain. With the development of deep learning technology, neural network-based algorithm models have achieved better results in both denoising and demosaicing tasks. With the support of a huge amount of training data and model parameters, it is possible to use neural networks to model multiple degradation types at the same time. Therefore, the joint denoising and demosaicing network models the two processes at the same time, which can prevent error accumulation and reuse image features at the same time. For example, the paper "Beyond Joint Demosaicking and Denoising: An Image Processing Pipeline for a Pixel-bin Image Sensor" at the 2021 IEEE / CVF Conference on Computer Vision and Pattern Recognition Workshops introduced multiple mechanisms such as attention modules and adversarial training to improve the joint denoising and demosaicing effects. However, although neural networks can learn the restoration process well, in dark light scenes with low signal-to-noise ratio, single image restoration algorithms have performance bottlenecks in noise removal and detail recovery.

[0003] On the other hand, with the development of multi-sensor devices, multi-image fusion processing has become an important means of overcoming the bottleneck of single-image algorithms. Monochrome-color binocular camera systems are widely used in devices such as smartphones. Compared to color cameras, monochrome cameras are not affected by color filters during the imaging process and have a higher light intake, thus offering significant advantages in image quality. Using the monochrome camera to guide the color camera for denoising and demosaicing can fully leverage the combined advantages of multiple sensors. However, existing guided restoration neural network methods cannot directly handle misaligned scenes. The paper "Robust Joint Image Reconstruction from Color and Monochrome Cameras," presented at the 2019 British Machine Vision Conference, first performs pixel-level registration of binocular images based on optical flow to mitigate the effects of parallax. The restoration results are then iteratively solved using traditional optimization methods. This method is computationally intensive and struggles to achieve accurate registration in low signal-to-noise ratio scenarios, further impacting the subsequent joint denoising and demosaicing processes. Summary of the Invention

[0004] Based on the imaging advantages of black and white cameras, the present invention proposes a black and white image guided joint denoising and demosaicing method for color RAW images. In order to eliminate the non-alignment factors caused by parallax between the two, the present invention constructs an alignment guidance image generation module based on the attention mechanism, and trains it based on the perceptual loss function, thereby generating high-quality alignment guidance images for guiding RAW image denoising and demosaicing. At the same time, the present invention trains the joint guided denoising and demosaicing network module based on the color-structure loss function, so that the restoration result has a better visual effect. By generating high-quality alignment guidance images, the method of the present invention can handle non-aligned guided joint denoising and demosaicing scenes containing parallax, while allowing the black and white image itself to be contaminated by noise without affecting the guidance effect. Based on the parallax attention mechanism, the influence of binocular parallax and the noise of the black and white camera itself is reduced, and the image information of the black and white camera in the black and white-color binocular system is used to assist the joint denoising and demosaicing process of the color camera.

[0005] The present invention provides a method for joint denoising and demosaicing of color RAW images guided by black and white images. The method utilizes high signal-to-noise ratio black and white camera images to guide cross-parallax color camera RAW images for joint denoising and demosaicing. The method comprises the following steps:

[0006] S1: Construct a noisy black-white-color binocular camera simulation dataset in dark light scenes.

[0007] S2: Build a joint denoising and demosaicing model based on the dataset in step S1.

[0008] The joint denoising and demosaicing model includes an alignment guide image generation module and a guided denoising and demosaicing module.

[0009] S21: Construct the alignment guidance image generation module, which solves the feature correlation between the black and white image and the RAW image in the disparity direction based on the disparity attention mechanism, and uses the relevant structural features to construct the alignment guidance image features, thereby obtaining the alignment guidance image; using the true value of the clean grayscale image corresponding to the RAW image as supervision, the alignment guidance image generation module is trained based on the perceptual loss function until convergence, so that the alignment guidance image generated by the alignment guidance image generation module is as close as possible in structure to the RAW image.

[0010] S22: Constructing the guided denoising and demosaicing module, the guided denoising and demosaicing module extracts the features of the RAW image and the features of the aligned guide image respectively, upsamples the feature resolution of the RAW image, fuses the features of the two by splicing feature channels, and finally generates a clean RGB image corresponding to the RAW image through feature decoding, completing the guided denoising and guided demosaicing processes at the same time; training the guided denoising and demosaicing module based on the structure-color loss function until convergence, so that the denoising and demosaicing results have both accurate color reconstruction and sharp detail structure.

[0011] The alignment guide image generation module fuses the non-aligned black and white camera image information with the color camera image information based on the parallax attention mechanism to generate a high-quality guide image aligned with the color camera; the guided denoising and demosaicing module guides the joint denoising and demosaicing process of the color camera RAW image based on the high-quality aligned guide image.

[0012] S3: Denoise and demosaic the color RAW image based on the joint denoising and demosaicing model constructed in step S2. For a black-and-white and color binocular camera system, images are simultaneously acquired in low light conditions. The alignment-guided image generation module leverages the additional high signal-to-noise ratio information provided by the black-and-white camera images and mitigates the effects of misalignment. This module guides the denoising and demosaicing processes of the color camera RAW image, producing visually appealing color low-light images.

[0013] Step S1 generates a large number of noisy black-and-white-and-color binocular system simulation datasets in dark light scenes based on the existing color binocular image dataset, which is used for training and testing the joint denoising and demosaicing model. The method for constructing the noisy black-and-white-and-color binocular system simulation dataset is as follows:

[0014] S11 divides the brightness of each pixel in the normal image captured by the binocular color camera by K to simulate a dark light scene.

[0015] For the left view, the S12 samples the color channel values ​​based on the RGGB Bayer pixel arrangement to generate a RAW image. For the right view, it adds the three color channel values ​​to simulate the full light input of a black and white camera to generate a black and white image.

[0016] S13 adds Poisson-Gaussian noise with the same parameters to the black and white image and the RAW image.

[0017] The construction of the joint denoising and demosaicing model in step S2 consists of two parts: one is to build an alignment guidance image generation module, and the other is to build a guided denoising and demosaicing module:

[0018] The alignment guidance image generation module solves the feature correlation between the black and white image and the RAW image in the disparity direction based on the disparity attention mechanism, and constructs the alignment guidance image features using the relevant structural features. The specific method is as follows:

[0019] First, the RAW image is demosaiced based on the traditional demosaicing algorithm and converted into a single-channel grayscale image by adding the three color channels. Secondly, the RAW image features F from the left and right views are extracted by the same feature extractor. raw With the black and white image feature F mono , reduce noise interference in the feature space and enhance structural information. Then, based on the parallax attention mechanism, raw Features and F mono The feature solves the correlation weight matrix M in the disparity direction and uses the correlation in the row direction to construct the alignment guidance image feature: Where f represents the convolution operation, To align the features of the guidance image, the feature information from the guidance black and white image is weighted and fused according to the disparity correlation. Finally, the feature decoder is used to The features are decoded into a high-quality aligned guidance image that is structurally consistent with the target color image.

[0020] The method to solve the correlation weight matrix M is: for the RAW image features F from the left and right views raw With the black and white image feature F mono, both tensors have dimensions of H×W×C, where H, W, and C represent the height, width, and number of channels of the feature tensor, respectively. First, the left view features are passed through a 3×3 convolutional layer to generate a query tensor, while the right view features are passed through two different 3×3 convolutional layers to generate a key tensor and a value tensor, each of which has dimensions of H×W×C. The dimensions of the key tensor are rearranged to convert the dimensions to H×C×W. The query tensor and the rearranged key tensor are matrix-multiplied to obtain a correlation calculation result matrix of dimension H×W×W. A SoftMax operation is then performed to obtain the correlation weight matrix M in the row direction.

[0021] The perceptual loss function is used as the loss function for training the alignment-guided image generation module. The perceptual loss function is defined as: Among them, L p represents the perceptual loss function; The output result G of the alignment guidance image generation module is the reference truth value used as supervision, which is set to the clean grayscale image truth value corresponding to the RAW image; F VGG (·) represents image features extracted using a pre-trained VGG model. This perceptual loss function calculates the distance between the output and the ideal value in the VGG feature space. By optimizing this distance during training, the generated alignment guidance images focus on structural reconstruction, making them more efficient.

[0022] The guided denoising and demosaicing module uses the generated high-quality aligned guide image to replace the non-aligned, noisy black-and-white camera image to guide the joint denoising and demosaicing of the color camera RAW image. The guided denoising and demosaicing module extracts features from the noisy RAW image and the high-quality aligned guide image separately, and fuses these features by splicing feature channels. The guided denoising and demosaicing module is trained based on a structure-color loss function and directly decodes the RAW image to generate a clean RGB image, completing both the guided denoising and demosaicing processes simultaneously.

[0023] The neural network mainly learns structural features from the aligned guidance image and color features from the RAW image, and uses the structure-color loss function as the loss function to guide the training of the joint denoising and demosaicing module. The structure-color loss function is defined as: Among them, L pc is the structure-color joint loss function; is the result of joint denoising and demosaicing; Y is the reference truth value, which is set to the true value of the clean RGB image corresponding to the noisy RAW image. The first term of the loss function is the structural loss, F VGG(·) means that the image features are extracted through the pre-trained VGG model, and the structural information of the output results is constrained by the VGG spatial features; the second term of the loss function is the color loss, F gaussian (·) represents the extraction of low-frequency information of the image through Gaussian filtering, and the color loss function is obtained by F gaussian The (·) function calculates the loss in the low-frequency space of the image. It constrains the denoising and demosaicing results from both structural and color aspects, so that the structure of the aligned guidance image is fully transferred while maintaining the color accuracy of the output result.

[0024] The beneficial effects of this invention include: using the parallax attention mechanism to eliminate misalignment in the guidance process, allowing the guidance information of the black-and-white camera image to be accurately assigned across parallax to the denoising and demosaicing process of the color camera. Furthermore, through the design and combination of the perceptual loss function and the structure-color loss function, accurate high-frequency detail structure is reconstructed after denoising, while the demosaicing process achieves accurate color interpolation. This invention has strong application value for binocular imaging devices in low-light scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a flow chart of the method of the present invention.

[0026] Figure 2 This is a workflow diagram of the alignment guidance image generation module of the present invention.

[0027] Figure 3 It is a structural diagram of the feature extractor of the present invention.

[0028] Figure 4 This is a schematic diagram of the network structure of the present invention that fuses the features of the left and right views based on the parallax attention mechanism.

[0029] Figure 5 This is a schematic diagram of the alignment-guided image generation principle based on the disparity attention mechanism. (a) is the target image visualization result, which is the output image of the left camera; (b) is the black and white image, which is the output image of the right camera. For the three rows of pixels marked L1, L2, and L3 in (a) and (b), the disparity attention solution results are shown in (c), (d), and (e).

[0030] Figure 6 It is a workflow diagram of the guided denoising and demosaicing module of the present invention.

[0031] Figure 7 This is a network architecture diagram of the guided denoising and demosaicing module of the present invention.

[0032] Figure 8This figure illustrates a specific embodiment of the present invention's process of generating an alignment guide image based on a black-and-white image and performing guided denoising and demosaicing. (a) is a simulated RAW image from a color camera under low light conditions, (b) is the visualized image of (a) after preliminary demosaicing and brightness stretching, (c) is the input black-and-white camera image, (d) is the generated alignment guide image, (e) is the generated denoising and demosaicing result, and (f) is the reference target image. DETAILED DESCRIPTION

[0033] The following is further described with reference to specific embodiments and drawings.

[0034] Example

[0035] First, based on the color binocular image dataset, a black and white-color binocular camera simulation dataset is constructed, specifically:

[0036] The normal image captured by the binocular color camera is divided by K in brightness to simulate a dark scene (the maximum pixel value of the dark scene image is less than 40). Subsequently, the color channel values ​​of the left view are sampled based on the RGGB Bayer pixel arrangement pattern to generate a Bayer RAW image. For the right view, the three color channel values ​​are added together to simulate the full amount of light entering the black and white camera to generate a black and white image. Finally, because in real scenes, both black and white and color cameras are inevitably contaminated by real noise, Poisson-Gaussian noise with the same parameters is added to both. This results in a noisy RAW-black and white binocular image pair in a dark scene, which is used for subsequent network training.

[0037] The network architecture proposed by the method of the present invention is as follows Figure 1 As shown in the figure, it is mainly divided into two modules: "alignment-guided image generation" and "guided denoising and demosaicing".

[0038] Alignment guide image generation module such as Figure 2 As shown in the figure, it uses the parallax attention mechanism to eliminate the misalignment between the black and white image and the RAW image, generating an aligned guidance image. At the same time, the module is trained separately with the perceptual loss as the loss function using the true black and white image value of the RAW image as supervision information until convergence.

[0039] The alignment-guided image generation process based on the parallax attention mechanism is described as follows:

[0040] First, for the left and right camera images, the same feature encoder is used for feature encoding. The specific structure of the feature encoder is as follows: Figure 3As shown in the figure, the RAW image is first demosaiced using the traditional demosaicing algorithm DDFAPD and converted into a single-channel grayscale image by adding the three color channels. The feature encoding process has two important functions: through feature encoding, the noise level of the input RAW image and the black-and-white image is suppressed, reducing interference with structural similarity judgment; and the target RAW image and the guide black-and-white image are converted to the same feature space, preserving their structural correlation.

[0041] Secondly, the RAW image features F from the left and right views are extracted by the same feature extractor raw With the black and white image feature F mono .

[0042] Then, the left and right view features are fused based on the parallax attention mechanism. The specific structure is as follows Figure 4 As shown in Figure 2. Since the non-alignment factors caused by parallax generally appear in the same row, the correlation between the left and right view features on the same row is calculated based on the parallax attention mechanism. raw Features and F mono The feature solves the correlation weight matrix M in the disparity direction and uses the correlation in the row direction to construct the alignment guidance image feature: Where f represents the convolution operation, To align the guided image features; finally, the feature decoder is used to The feature decoding is an alignment guide image that is consistent with the target color image structure. The alignment guide image features are linearly combined based on the correlation of the black and white image feature values ​​on the corresponding rows.

[0043] The solution method for the correlation weight matrix M is: for the RAW image features F from the left and right views raw With the black and white image feature F mono , both tensors have dimensions of H×W×C, where H, W, and C represent the height, width, and number of channels of the feature tensor, respectively. First, the left view features are passed through a 3×3 convolutional layer to generate a query tensor, while the right view features are passed through two different 3×3 convolutional layers to generate a key tensor and a value tensor, each of which has dimensions of H×W×C. The dimensions of the key tensor are rearranged to convert the dimensions to H×C×W. The query tensor and the rearranged key tensor are matrix-multiplied to obtain a correlation calculation result matrix of dimension H×W×W. A SoftMax operation is then performed to obtain the correlation weight matrix M in the row direction.

[0044] The feature correlation visualization results of the parallax attention mechanism are as follows Figure 5 shown. Figure 5 (a) is the left camera target image visualization result, Figure 5 (b) is the black and white image of the right camera. Figure 5 (a) Figure 5 The feature correlation results of the three rows of pixels marked L1, L2, and L3 in (b) are as follows Figure 5 (c) Figure 5 (d) Figure 5 (e) shows that the feature correlation matrix forms a clear peak in the diagonal direction, indicating that the dislocation structure caused by disparity is well captured by the disparity attention mechanism.

[0045] Finally, the reconstructed features are decoded to generate an alignment guide image. In order to generate an alignment guide image with less noise and clear structure, the algorithm uses the true grayscale image corresponding to the RAW image as training supervision and uses perceptual loss as the loss function to train the module. The perceptual loss function is defined as: Among them, L p represents the perceptual loss function; The output result G of the alignment guidance image generation module is the reference truth value used as supervision, which is set to the clean grayscale image truth value corresponding to the RAW image; F VGG (·) represents image feature extraction using a pre-trained VGG model. The perceptual loss function optimizes the alignment and guidance of the image generation module by calculating the distance in the feature space of the pre-trained VGG network, making the generated results more consistent in structure and more visually pleasing to the human eye.

[0046] The guided denoising and demosaicing module is as follows Figure 6 As shown in the figure, it extracts RAW image features and alignment guide image features respectively, and after upsampling the RAW image feature resolution, it fuses the information of the two by feature splicing, and finally outputs the denoising and demosaicing results through feature decoding. Based on the structure-color loss function, the module is trained so that the denoising and demosaicing results have both accurate color reconstruction and sharp detail structure. The specific network structure of this module is shown in the figure. Figure 7 shown.

[0047] The structure-color loss function is defined as follows:

[0048]

[0049] Among them, L pv Represents the structure-color loss function, the first term is the perceptual loss function, and the second term is the color loss function. VGG (·) indicates that the image features are extracted by the pre-trained VGG model, and the perceptual loss function is obtained by F VGG (·) function calculates the loss in the deep feature space of the image. gausdian (·) represents the extraction of low-frequency information of the image through Gaussian filtering, and the color loss function is obtained by F gaussianThe (·) function computes the loss in the low-frequency space of the image. This joint loss function constrains the various structures of the image content in the VGG feature space, while constraining the overall brightness of the image content in the low-frequency space. This combination helps fully transfer the various structures in the guide image to the target image during training, while preventing the influence of the guide image's brightness on color reconstruction. This results in guided denoising and demosaicing with clear structure and accurate color.

[0050] Figure 8 is the operating result of a specific embodiment based on the method of the present invention, wherein, Figure 8 (d) is the alignment guide image generated by the method of the present invention, which has less noise, clear structure, and is consistent with the target color image structure. Figure 8 (e) is the denoising and demosaicing result generated by the method of the present invention, and Figure 8 (f) Compared with the reference target image, the structure and color are relatively consistent.

[0051] This paper proposes a novel binocular camera-guided denoising and demosaicing method. It generates high-quality aligned guided images in non-aligned scenes with parallax, and achieves visually pleasing denoising and demosaicing results based on a structure-color loss function. When deployed in a black-and-white-and-color binocular system, the method leverages the black-and-white camera's advantage of being unaffected by color filters to assist the color camera's demosaicing process. Furthermore, the black-and-white camera's advantage of receiving greater light during simultaneous binocular imaging aids the color camera's denoising process.

Claims

1. A method for joint denoising and demosaicing of color RAW images guided by black and white images, characterized in that: Using the black and white camera output image, cross-parallax guided color camera RAW image is jointly denoised and demosaiced. The specific steps are as follows: S1: Dataset construction, constructing a noisy black-white-color binocular camera simulation dataset in dark light scenes; S2: constructing a joint denoising and demosaicing model based on the data set in step S1; the joint denoising and demosaicing model includes an alignment guide image generation module and a guide denoising and demosaicing module; First, the alignment guidance image generation module is constructed. The alignment guidance image generation module solves the feature correlation of the black and white image and the RAW image in the disparity direction based on the disparity attention mechanism, and constructs the alignment guidance image features using the relevant structural features, thereby obtaining the alignment guidance image; Using the true value of the clean grayscale image corresponding to the RAW image as supervision, the alignment guided image generation module is trained to convergence based on the perceptual loss function; Secondly, the guided denoising and demosaicing module is constructed. The guided denoising and demosaicing module extracts the features of the RAW image and the features of the aligned guide image respectively, and performs feature fusion by splicing feature channels. Finally, a clean RGB image corresponding to the RAW image is generated through feature decoding, completing the guided denoising and guided demosaicing processes simultaneously. The guided denoising and demosaicing module is trained until convergence based on the structure-color loss function. S3: Denoise and demosaic the color RAW image based on the joint denoising and demosaicing model constructed in step S2.

2. The method for joint denoising and demosaicing of color RAW images guided by black and white images according to claim 1, characterized in that: The step S1 specifically includes the following steps: 1.1 Divide the brightness of each pixel in the normal image captured by the binocular color camera by K to simulate a dark light scene; 1.2 For the left view, the color channel values ​​are sampled based on the RGGB Bayer pixel arrangement pattern to generate a RAW image. For the right view, the full light input of a black and white camera is simulated by adding the values ​​of the three color channels to generate a black and white image. 1.3 Add Poisson-Gaussian noise pollution with the same parameters to the black and white image and the RAW image.

3. The method for joint denoising and demosaicing of color RAW images guided by black and white images according to claim 1, characterized in that: In step S2, the alignment guide image generation module solves the feature correlation between the black and white image and the RAW image in the disparity direction based on the disparity attention mechanism, and constructs the alignment guide image features using the relevant structural features. The specific method is as follows: First, the RAW image is demosaiced based on the traditional demosaicing algorithm and converted into a single-channel grayscale image by adding the three color channels; secondly, the RAW image features from the left and right views are extracted using the same feature extractor. Black and white image features ; Then, based on the parallax attention mechanism, Features and Features solve the correlation weight matrix in the disparity direction , and use the correlation in the row direction to construct the alignment guidance image feature: ,in represents the convolution operation, To align the guided image features; finally, the feature decoder is used to The features are decoded into an aligned guidance image that is structurally consistent with the target color image.

4. The method for joint denoising and demosaicing of color RAW images guided by black and white images according to claim 3, characterized in that: In step S2, the correlation weight matrix The solution is: For RAW image features from left and right views Black and white image features , the tensor dimensions of both are H×W×C, where H, W, and C represent the height, width, and number of channels of the feature tensor respectively; first, the left view features are passed through a 3×3 convolution layer to generate a query tensor, while the right view features are passed through two unshared 3×3 convolution layers to generate a key tensor and a value tensor, all of which have dimensions of H×W×C; the dimensions of the key tensor are rearranged to convert the dimensions to H×C×W; the query tensor and the rearranged key tensor are matrix multiplied to obtain a correlation calculation result matrix of dimension H×W×W, and the correlation weight matrix in the row direction is obtained through SoftMax operation .

5. The method for joint denoising and demosaicing of color RAW images guided by black and white images according to claim 1, wherein: In step S2, the perceptual loss function is defined as: ,in, represents the perceptual loss function; represents the output result of the alignment guidance image generation module; Represents the reference truth value used as supervision, which is set to the clean grayscale image truth value corresponding to the RAW image; Indicates that image features are extracted through the pre-trained VGG model.

6. The method for joint denoising and demosaicing of color RAW images guided by black and white images according to claim 1, wherein: In step S2, the structure-color loss function is defined as: ,in, is the structure-color joint loss function; This is the joint denoising and demosaicing result; is the reference truth value, which is set to the clean RGB image truth value corresponding to the noisy RAW image; Indicates that image features are extracted through the pre-trained VGG model; Indicates extracting low-frequency information of the image through Gaussian filtering.

Images