A perceptually friendly and noise-reducing-intensity-controllable method, apparatus, and electronic equipment for joint denoising and de-mosaicing of RAW images
By employing a perception-friendly and denoising-intensity-controllable joint denoising and desacrifice method, the problem of fixed denoising behavior in RAW images is solved, enabling continuous adjustment of denoising intensity and adaptation to user preferences, thereby improving image restoration quality and efficiency.
Patent Information
- Application Number
- CN202611429222.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-15
- Publication Date
- 2026-11-10
AI Technical Summary
Existing methods for joint denoising and desacrifice of RAW images have fixed denoising behavior, which leads to detail smearing and artifacts. They are difficult to adapt to different user preferences and spatial non-uniform degradation, and fail to effectively distinguish between luminance noise and chrominance noise that affect perceived quality.
We employ a perception-friendly and noise reduction method with controllable noise reduction intensity. By using a controllable joint noise reduction and noise reduction network and a perception adapter, we can continuously adjust the noise reduction intensity, prioritize the suppression of chromatic noise, adapt to spatial non-uniform degradation, and use noise priors to generate spatially adaptive convolutional kernels.
It achieves continuous adjustment of denoising intensity, reduces artifacts and white points, improves the user's subjective perception quality, adapts to the recovery needs of different locations, and reduces computational load and inference latency.
Smart Images

Figure CN123071184A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of digital image processing, computational photography, and camera image signal processing, and particularly to a perception-friendly and noise reduction method, apparatus, and electronic device for Bayer array RAW images with controllable noise reduction intensity. This invention is applicable to image signal processing pipelines for mobile terminals, digital cameras, and smart video devices. Background Technology
[0002] Digital camera sensors typically acquire single-channel RAW data via color filter arrays. Before forming a displayable image, the RAW data needs to undergo denoising, demosaicing, white balance, color correction, tone mapping, and gamma correction. Denoising suppresses random disturbances caused by photon shot noise, readout noise, etc., while demosaicing recovers missing color components based on Bayer sampling locations. If denoising and demosaicing are processed sequentially by independent modules, residual errors from the earlier stages are amplified by interpolation in the later stages, and the combined deployment of multiple modules increases system complexity. Therefore, joint denoising and demosaicing has become an important solution for early-stage RAW image restoration.
[0003] Existing joint denoising and de-mosaicing methods typically use paired degraded RAW images and clean target images for supervised training, with the estimated noise level map used as a network condition. This training approach tends to teach the network fixed denoising behavior: in low-light conditions, high gain, or when model capacity is limited, the network tends to over-denoise in pursuit of high pixel fidelity, resulting in the blurring of hair, skin texture, leaves, and fine lines; simply reducing the noise level map to weaken denoising produces severe artifacts in existing models, failing to stably achieve a balance between noise suppression and detail preservation. Meanwhile, different users have different preferences for smoothness, graininess, and realism, making it difficult to meet the needs of interactive imaging with a fixed denoising intensity.
[0004] Furthermore, the human visual system is generally more sensitive to chromatic noise than to luminance noise of the same magnitude. Existing tunable methods, if they do not distinguish between luminance and chromatic noise, may retain unpleasant color noise while preserving details. A direct solution is to add a chromatic denoising network in the sRGB domain to improve perceptual quality, but this creates a two-level inference chain between the RAW and sRGB domains, increasing latency, computational cost, and deployment complexity. Moreover, shot-readout noise in the RAW domain exhibits signal correlation and significant spatial differences, and demosaicing in textured areas is more difficult than in flat areas. Using fixed convolutional kernels that are independent of content and degradation makes it difficult to simultaneously address the recovery needs of different locations.
[0005] Therefore, there is an urgent need for a joint denoising and desacrifice scheme for RAW images that can achieve continuous adjustment of denoising intensity through a single joint denoising and desacrifice network, maintain physical consistency in desacrifice supervision while reducing denoising, prioritize the suppression of chroma noise, and adaptively process spatial non-uniform degradation. Summary of the Invention
[0006] This invention aims to address the problems in existing RAW image joint denoising and demosaicing techniques, such as fixed denoising behavior, excessive denoising leading to detail smearing, artifacts caused by directly reducing noise conditions, chromatic noise affecting perceptual quality, and the difficulty of fixed convolution kernels adapting to spatial non-uniform degradation. It provides a perceptually friendly joint denoising and demosaicing method, apparatus, and electronic device with controllable denoising intensity, enabling users to continuously adjust the balance between noise suppression and detail preservation during the inference stage by controlling factors, while maintaining high color reconstruction quality and inference efficiency.
[0007] To achieve the above objectives, the present invention provides the following technical solution:
[0008] Perceptually friendly and noise reduction intensity controllable joint denoising and de-mosaic methods for RAW images include:
[0009] First training phase: Obtain clean sRGB images, and obtain clean RAW image R*, degraded RAW image R, and clean linear RGB image through reverse processing of a differentiable bidirectional image signal processing pipeline and degradation simulation. And noise level map M; perform pixel inversion on the degraded RAW image R and the noise level map M to obtain a four-channel image. and Sampling denoising control factor And branch random variable s, using the denoising control factor Construct input conditions with a mismatch between noise level and denoising intensity, and training targets containing a specified proportion of noise residuals. Combine this with supervision from regular clean targets to train a controllable joint denoising and desacrifice network. ;
[0010] Second training phase: Freeze the controllable joint denoising and de-mosaicing network obtained in the first training phase and use it as the teacher network. The perceptual training objective is constructed using the luminance channel output by the teacher network and the chroma channel of a clean image; the student network is initialized and frozen using the weights of the teacher network. Training the perceptual adapter connected to the student network. ;
[0011] Inference phase: Acquire the real degraded RAW image and its noise level map, and receive the denoising control factor set by the user or the system. The degraded RAW image and the processed image A scaled noise level map is input into the trained student network and the perception adapter, and the output is a restored linear RGB image.
[0012] Furthermore, in the first training phase, the input condition of constructing a noise level that does not match the denoising intensity and the training objective containing a specified proportion of noise residuals include: when At that time, The noise condition input is used as the noise condition input to the controllable joint denoising and demosaicing network, and the noise residual of the degraded RAW image relative to the clean RAW image is used. proportionally Inject a clean, linear RGB target at the Bayer sampling domain; when At that time, an unscaled noise level map was used. and clean linear RGB images Implement routine paired monitoring.
[0013] Furthermore, in the input condition where the constructed noise level and denoising intensity do not match, and the training objective containing a specified proportion of noise residuals, when When the training objective is expressed as:
[0014] and Minimize the differences between them;
[0015] when When the training objective is expressed as:
[0016] and Minimize the differences between them;
[0017] in, This represents the mosaic operator that projects a three-channel image onto the Bayer sampling location.
[0018] Furthermore, in the second training phase, constructing the perceptual training target using the luminance channel output by the teacher network and the chroma channel of the clean image includes:
[0019] The teacher network The output is processed through the forward image signal processing flow. Mapped to the sRGB domain, then converted to the YUV domain, the teacher's luminance channel is obtained. Teacher color channel , ;
[0020] Clean sRGB image Convert to the YUV domain to obtain , , ;
[0021] Using the teacher brightness channel The chroma channels of the clean image , splicing together to form a perceptual training objective .
[0022] Furthermore, the sensing adapter The student network encoder and the cropped copy of the intermediate block, whose inputs include , and Output control features The control features are injected into the corresponding layer of the student network via zero-initialization convolution, so that the student output is consistent with the teacher output at the start of training.
[0023] Furthermore, the controllable joint denoising and desmosaic network adopts a U-shaped multi-scale encoder-intermediate layer-decoder structure, in which a degradation-aware convolutional layer (DAConv) is incorporated; the degradation-aware convolution extracts the noise prior from the noise level map. With degenerative characteristics After concatenation along the channel dimension, spatial adaptive weights are generated sequentially through 1×1 pointwise convolution, depthwise convolution, and 1×1 pointwise convolution. The weights are rearranged into grouped convolutional kernels, and dynamic convolution is performed on the neighborhood centered at each position, outputting the result. Add the residuals together.
[0024] Furthermore, the spatial adaptive weights The generation method is represented as follows:
[0025] ;
[0026] Wherein, Concat represents concatenation along the channel dimension, PConv represents 1×1 pointwise convolution, and DConv7 represents 7×7 depthwise convolution.
[0027] Furthermore, the differentiable bidirectional image signal processing pipeline includes at least one reversible process among mosaic / demosaic, exposure compensation / white balance, color correction, tone mapping, and gamma correction; the degradation simulation includes injecting shot noise and readout noise into the clean RAW image, and the noise level map... Represented as:
[0028] ;
[0029] in, Indicates the total gain. This represents the variance of the readout noise.
[0030] Furthermore, the first training phase employs hard logarithmic loss. Optimize the controllable joint denoising and desacrifice network:
[0031] ;
[0032] in, , , These represent the number of channels, height, and width, respectively, and ε is a stability constant.
[0033] The present invention also provides a perception-friendly and noise reduction intensity-controllable RAW image joint denoising and de-mosaicing device, comprising:
[0034] The data construction module is used to generate clean RAW images, degraded RAW images, clean linear RGB images, and noise level maps through differentiable bidirectional image signal processing and degradation simulation.
[0035] A controllable training module is used to scale the noise level map according to the denoising control factor and switch between the Bayer domain noise residual target and the three-channel clean target to train a controllable joint denoising and desacrifice network.
[0036] The perception adaptation module is used to generate perception targets using the teacher's luminance channel and clean chrominance channel, and to train the perception adapter connected to the frozen student network.
[0037] The degradation-aware restoration module is used to generate spatially adaptive convolution kernels based on noise priors and local content, and jointly perform RAW image denoising and de-mosaicing.
[0038] The intensity control and output module receives control factors, inputs the degraded RAW image and the scaled noise level map into the trained network, and outputs an sRGB image after image signal processing.
[0039] The present invention also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described above.
[0040] Compared with the prior art, the present invention has at least the following beneficial effects:
[0041] (1) By scaling the noise level map simultaneously during training and injecting a specified proportion of real RAW noise residuals into the target, the denoising intensity of a single network can be continuously adjusted during the inference phase, avoiding repeated training or deployment of multiple models for different styles.
[0042] (2) Project the supervision containing the noise residuals onto the Bayer sampling location and use it alternately with the three-channel clean target supervision to reduce white spots, zipper marks and color deviations introduced by pre-de-mosaic while preserving noise and texture;
[0043] (3) Using teacher luminance and clean chroma to construct YUV domain perception targets, and using a lightweight perception adapter to suppress chroma noise while retaining luminance particles and details, thereby improving the user's subjective perception quality;
[0044] (4) Utilize noise prior and local content to generate spatially adaptive grouped convolution kernels to enhance the processing capabilities for RAW domain signal-related noise, spatial non-uniform degradation, and demosaicing of complex texture regions.
[0045] (5) By using a one-stage student network and lightweight adapter, pointwise convolution and depthwise convolution, controllable recovery capability is achieved while maintaining low parameter quantity, computational quantity and inference latency, making it suitable for high-resolution image and mobile terminal deployment. Attached Figure Description
[0046] Figure 1 This is the overall flowchart of an embodiment of the present invention, wherein (a) is training phase I: training of a controllable joint denoising and de-mosaic network, (b) is training phase II: training of a perceptron adapter, and (c) is the inference process;
[0047] Figure 2 This is a schematic diagram of the network structure and degradation-aware convolution of an embodiment of the present invention, where (a) is the coding block structure and (b) is the degradation-aware convolution DAConv structure. Detailed Implementation
[0048] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the following embodiments are only used to explain the present invention and are not intended to limit the scope of protection; in the absence of conflict, the various embodiments and their technical features can be combined with each other.
[0049] 2.1 Overall Method Flow (Two-stage training + Controlled inference)
[0050] like Figure 1 As shown, the present invention provides a method for joint denoising and de-mosaicing of RAW images, which includes a first training phase, a second training phase, and an inference phase.
[0051] A) First training phase: Obtain clean sRGB images, and obtain clean RAW images through reverse processing of a differentiable bidirectional image signal processing pipeline and degradation simulation. Degraded RAW images Clean linear RGB image Noise Level Chart ;right and Perform pixel inversion to obtain four channels. and Sampling denoising control factor and branching random variables ,use Construct input conditions with a mismatch between noise level and denoising intensity, and training targets containing a specified proportion of noise residuals. Combine this with supervision from regular clean targets to train a controllable joint denoising and desacrifice network. .
[0052] B) Second training phase: Freeze the controllable joint denoising and de-mosaicing network obtained in the first training phase and use it as the teacher network. The perceptual training objective is constructed using the luminance channel of the teacher's output and the chroma channel of a clean image; the student network is initialized and frozen using teacher weights. Lightweight sensor adapters for training and connecting to student networks This enables the student network to suppress chroma noise based on the inherited controllable joint denoising and mosaic prior.
[0053] C) Inference Phase: Acquire the real degraded RAW image and its noise level map, and receive the denoising control factor set by the user or the system. Degraded RAW images and processed images The scaled noise level map is input to the trained student network and sensor adapter, and the output is a reconstructed linear RGB image, which is then processed through a real camera image signal processing flow to obtain an sRGB image. Control factor A smaller value results in stronger noise reduction and smoother output; control factor Larger sizes retain more luminance noise and fine-grained texture.
[0054] 2.2 Differentiable bidirectional image signal processing and degradation simulation
[0055] The differentiable bidirectional image signal processing pipeline includes at least one reversible process among mosaic / demosaic, exposure compensation / white balance, color correction, tone mapping, and gamma correction. The inverse pipeline is used to synthesize a clean RAW image and an intermediate clean linear RGB image from a clean sRGB image, while the forward pipeline maps the network output to the standard sRGB domain. Camera-related parameters can be acquired from metadata of various devices and randomly sampled or interpolated during training.
[0056] In one implementation, exposure compensation employs the following forward and inverse relationships:
[0057] (1)
[0058] White balance can use a red channel gain of 0.9 to 3.1 and a blue channel gain of 1.2 to 4.2; color correction can obtain multiple 3×3 color correction matrices from camera metadata and perform random sampling and interpolation; tone mapping can construct differentiable forward and inverse mappings based on multiple representative tone curves obtained from camera metadata through uniform grid linear interpolation; gamma correction can use... and its inverse mapping The above range is for illustrative purposes only and not a limitation.
[0059] Degradation simulation involves injecting shot noise and readout noise into the clean RAW image and can add lens blur in the linear RGB stage of the reverse pipeline. Degraded RAW image It can be represented as:
[0060] (2)
[0061] in, Represents the Poisson distribution. Indicates a Gaussian distribution. Indicates the total gain. This represents the variance of the readout noise. Sampling is possible in the logarithmic domain. and according to Follow the mean variance is Use a Gaussian distribution to sample readout noise; for example Sampling can be performed between 0.0001 and 0.04. Sampling is possible between 1.0 and 3.0. Sampling can be performed between -0.5 and 1.6. A value of 0.3 is acceptable. For lens blurring, an isotropic Gaussian kernel with a kernel size of 5 and a variance of 0.5 can be used.
[0062] Based on the Gaussian approximation of the Poisson distribution, degraded RAW images can be approximated by a mean of . variance is Gaussian distribution, noise level graph It can be represented as:
[0063] (3)
[0064] Clean RAW images in practical applications When unavailable, degraded RAW images can be used. Alternative Estimated noise level map. Then... and Perform pixel inversion to package the 2×2 sampling units of the Bayer array into four channels. and .
[0065] 2.3 Controllable Joint Denoising and Depixelation Training
[0066] like Figure 1 As shown, the first training phase uses a noise reduction control factor. The explicit modeling of noise levels results in a mismatch between the desired denoising intensity and the actual noise level. Each training sample is sampled from random variables. and using threshold Switch between the two supervision branches.
[0067] when At that time, As a noise condition for the network, the desired denoising intensity received by the network is lower than the actual noise in the image; simultaneously, the noise residual of the degraded RAW image relative to the clean RAW image is considered. proportionally Inject a clean, linear RGB target. Because... Located in the Bayer sampling domain, and with clean linear RGB for three channels, the mosaic operator is first used. The clean linear RGB is projected onto the Bayer sampling location and then added to the noise residual, thus avoiding edge artifacts and color biases introduced by pre-interpolation noise using the demosaicing algorithm. The objective of this branch can be expressed as: .
[0068] when At that time, an unscaled noise level map was used. and clean linear RGB images Performing regular pairwise supervision allows the network to acquire explicit knowledge of missing color component reconstruction. By alternating between these two types of supervision, the network can learn both a specified proportion of noise preservation and correct demosaicing.
[0069] In one implementation, hard logarithmic loss is used. An optimized and controllable joint denoising and desacrifice network is constructed, wherein the loss function assigns a higher weight to low-value specular correlation pixel errors in the RAW domain, and its form is as follows:
[0070] (4)
[0071] The controllable joint denoising and demosaic loss can be written as:
[0072] like
[0073] (5)
[0074] like
[0075] (6)
[0076] in, , , These are the number of channels, height, and width, respectively. Let be a stability constant. During training, we can set . Follow the interval Uniform distribution on Follow the interval Uniform distribution on We set it to 0.5. Because the training process covers multiple scaled noise level maps, during inference... It can be adjusted within the training range, or extended beyond the training range based on image content and user preferences. Optionally, for... Set upper and lower limits to avoid negative noise conditions or insufficient noise reduction.
[0077] 2.4 Perceptual adapters inspired by the human visual system
[0078] like Figure 1 As shown, the second training phase aims to ensure that the control factors primarily retain luminance noise and fine-grained texture, while suppressing chrominance noise, which is more sensitive to visual perception. The controllable joint denoising and desacrifice network is first frozen. And as a teacher network The teacher network output is processed through a forward image signal processing flow. Mapped to sRGB domain, then converted. Mapping to the YUV domain yields the teacher luminance channel. Teacher color channel , Clean sRGB image Convert to the YUV domain to obtain , , ;use and , splicing together to form a perceptual training objective :
[0079] (7)
[0080] Initialize the student network with teacher network weights And freeze its basic parameters; construct a perception adapter. The perception adapter is a cropped copy of the student network encoder and the intermediate block, and its input includes... , and Output control features Control features are injected into the corresponding layers of the student network via zero-initialization convolutions, ensuring that the student output matches the teacher output at the start of training, and gradually learning chroma noise modulation during training. The student output can be represented as:
[0081] (8)
[0082] Sensing adapter use The loss is optimized as a perceived loss:
[0083] (9)
[0084] The second training phase only updates the parameters of the perceptual adapter and its connection layers, thus retaining the controllable joint denoising and de-mosaic capabilities obtained in the first training phase with lower training and inference overhead. The above one-stage student network plus adapter scheme can be replaced by an independent sRGB domain chroma denoising network to form a two-stage inference, but the one-stage scheme has the advantages of lower computational cost and easier deployment.
[0085] 2.5 Joint Denoising and Demosaic Network with Degradation-Aware Convolution
[0086] like Figure 2 As shown, the joint denoising and demosaicing network can adopt a U-shaped multi-scale encoder-intermediate layer-decoder structure, containing four encoding layers, one intermediate layer, and four decoding layers. The encoding layers can contain 1, 2, 4, and 4 encoding blocks respectively, the intermediate layer contains 4 intermediate blocks, and each decoding layer contains one decoding block. At the input, 3×3 convolutions are used to extract degraded image features and noise prior features, respectively. Multi-scale features are obtained through downsampling, and the encoded features are passed to the decoding layer of the same scale through skip connections. The main body of the network mainly uses pointwise convolution and depthwise convolution to reduce the number of parameters and computational cost.
[0087] Degradation-aware convolution DAConv is set in the encoding block to inject noise priors extracted from the noise level map into the features. For degradation features... and noise prior First, the layers are concatenated along the channel dimension, and then spatially adaptive weights are generated by sequentially performing 1×1 pointwise convolution, 7×7 depthwise convolution, and 1×1 pointwise convolution. :
[0088] (10)
[0089] To reduce computational load, Divided along the channel dimension into Groups, each group contains Each channel; the weight vector at each spatial location is rearranged into... Each 3×3 convolution kernel is shared among channels in the same group. For positions... Perform dynamic convolution on the 3×3 neighborhood centered at the center, and obtain , and then with Perform residual summation and output. Therefore, the convolution kernel changes simultaneously with the local image content and noise priors, allowing for different recovery behaviors for signal-related noise, flat regions, and textured regions.
[0090] 2.6 Inference Control and Optional Automatic Control
[0091] During the inference phase, noise level maps are estimated for real degraded RAW images. And perform pixel inversion; receive control factors ,Will and Input the student network and sensor adapter. Control factors can be provided via sliders in the graphical interface, camera shooting mode, scene recognition module, or preset styles. For noisy images, a smaller control factor can be used. To achieve stronger noise reduction; for images with less noise or rich texture, a larger [size / size] can be used. To preserve detail. Optionally, the content-aware control module may automatically predict based on estimated noise, texture intensity, exposure value, or user history. It can also generate control charts of spatial variation to apply different noise reduction intensities to different regions.
[0092] Example 1: Training Sample Construction and Controllable Joint Denoising and De-mosaic Network Training
[0093] like Figure 1 , Figure 2 As shown in the figure, this embodiment provides an implementation process for the first training phase.
[0094] S101: Obtain high-quality, clean sRGB images. Training images can be selected from the Waterloo ExplorationDatabase, DIV2K, and Flickr2K, or high-quality images taken with a camera can be used. Images are randomly cropped, flipped, and rotated, and pixels are normalized to a certain value. .
[0095] S102: Perform inverse image signal processing. This involves sequentially performing inverse gamma correction, inverse tone mapping, inverse color correction, inverse white balance, inverse exposure compensation, and Bayer mosaic to obtain a clean RAW image. and a clean linear RGB image in the middle During training, camera color correction matrices, tone curves, exposure values, and white balance gains are randomly selected to cover different camera imaging conditions.
[0096] S103: Synthesize the degraded RAW image. Poisson shot noise and Gaussian readout noise are added according to equation (2), and lens blur is simulated using a 5×5 isotropic Gaussian kernel in the linear RGB stage to obtain the degraded RAW image. Noise level map is generated according to equation (3). When a clean RAW file is not actually available, use Approximate calculation .
[0097] S104: Perform pixel inversion. Arrange pixels according to the Bayer array 2×2 unit. and Packaged into four channels respectively and And spliced along the channel dimension as network input.
[0098] S105: Generate controllable supervision. Sample each sample. , ,set up .when At that time, with and For network conditions, with For Bayer domain training objectives; when At that time, with and For network conditions, with The training objective is a three-channel linear RGB array.
[0099] S106: Construct a joint denoising and de-mosaic network. Employ methods such as... Figure 2 The diagram shows a four-layer encoder, a one-layer intermediate layer, and a four-layer decoder. The encoding blocks are configured with degradation-aware convolutions, and the basic channel width can be set according to the deployed computing power. Each layer forms a multi-scale recovery through downsampling, upsampling, and skip connections.
[0100] S107: Optimize the first-stage model. The loss functions from equations (4) to (6) and the Adam optimizer are used to update the parameters of the controllable joint denoising and de-mosaicing network. An example training setup is: cropped size 256×256, batch size 48, initial learning rate... The learning rate is halved at the 100,000, 150,000, 250,000, and 350,000 iterations, respectively. After training, the teacher network is obtained. .
[0101] Example 2: Perceptual Adapter Training and Controllable Inference
[0102] like Figure 1 As shown, this embodiment performs the second stage of training based on the teacher network obtained in Embodiment 1.
[0103] S201: Freezing the Teachers' Network .Will , and Input the teacher network and output a linear RGB image; use a forward image signal processing flow to map it to the sRGB domain, and then convert it to the YUV domain to obtain... .
[0104] S202: Construct the target for perception. Convert the corresponding clean sRGB image to the YUV domain to obtain... Preserve the brightness channel of the teacher's output. and with clean images , Replace teacher output , ,structure This objective does not forcibly remove all graininess from the teacher's luminance channel, thus preserving texture while suppressing color noise.
[0105] S203: Initialize the student network and perception adapter. Copy the teacher weights to the student network. And freeze; trim and copy the encoder and intermediate block to form a lightweight sensing adapter. Set a zero-initialized convolutional connection between the adapter output and the corresponding layer of the student network to make the initial control features zero.
[0106] S204: Training the sensor adapter. Perform forward image signal processing and YUV conversion on the student output, using equation (9). The loss constraint ensures that the student output is close to the perceived target, and only the perception adapter and connection layer are updated. Training can use the same clipping size, batch size, optimizer, and learning rate decay settings as in Example 1.
[0107] S205: Perform controlled inference. Acquire real camera RAW images and camera metadata, estimate and package the noise level map; determine the noise level using a slider in the user interface or a preset mode. ;Will and Input the student network and sensor adapter to obtain the linear RGB restoration result; perform forward image signal processing using the white balance, color correction, tone mapping and gamma parameters of the real camera to output the sRGB image.
[0108] Example 3: Devices, Electronic Devices, and Storage Media
[0109] 1) A perception-friendly and noise reduction intensity-controllable RAW image joint denoising and desacrifice device, comprising:
[0110] — Data construction module, used to generate clean RAW images, degraded RAW images, clean linear RGB images and noise level maps through differentiable bidirectional image signal processing and degradation simulation;
[0111] — A controllable training module is used to scale the noise level map according to the denoising control factor and switch between the Bayer domain noise residual target and the three-channel clean target to train a controllable joint denoising and desacrifice network.
[0112] — Perception adapter module, used to generate perception targets using the teacher luminance channel and clean chrominance channel, and train the perception adapter connected to the frozen student network;
[0113] — Degradation-aware restoration module, used to generate spatially adaptive convolution kernels based on noise priors and local content, and jointly perform RAW image denoising and de-mosaicing;
[0114] — The intensity control and output module receives control factors, inputs the degraded RAW image and the scaled noise level map into the trained network, and outputs an sRGB image after image signal processing.
[0115] 2) An electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the program, when executed, implements the above method.
[0116] 3) A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above method.
[0117] Experimental verification
[0118] To verify the present invention, high-quality images from the Waterloo Exploration Database, DIV2K, and Flickr2K were used for training. Test data included real RAW images captured by OPPO Find X7 Ultra, iPhone 13 Pro, Xiaomi 12, and Huawei Mate 30, as well as publicly available DND benchmarks. For smartphone images without a ground truth reference, MUSIQ, HyperIQA, and MANIQA were used for evaluation; the DND benchmarks used PSNR, SSIM, and no-reference perception metrics for evaluation.
[0119] (1) On the DND criterion, this method is The proposed method achieves PSNR of 39.986 dB, SSIM of 0.9606, HyperIQA of 0.3642, and MANIQA of 0.2865 in sRGB values. In comparison, TENet++ achieves PSNR of 39.733 dB and SSIM of 0.9559, while JDSR achieves PSNR of 39.574 dB and SSIM of 0.9527. These results demonstrate that the proposed method maintains controllability while achieving high accuracy in joint denoising and de-mosaicing.
[0120] (2) In the ablation experiment on Xiaomi 12 data, the MUSIQ, HyperIQA, MANIQA, and PaQ2PiQ of the complete protocol were 54.833, 0.4641, 0.2943, and 70.282, respectively; after removing the Bayer domain mosaic projection, they were 54.641, 0.4439, 0.2911, and 70.404, respectively; after removing... After removing the mosaic overlay from the three channels, the values are 53.770, 0.4396, 0.2880, and 70.126, respectively; after removing the perceptual adapter, they are 53.674, 0.4474, 0.2917, and 70.168, respectively; and after removing the degenerate perceptual convolution, they are 53.356, 0.4256, 0.2834, and 70.013, respectively. These results validate the roles of the two types of supervision, the perceptual adapter, and the degenerate perceptual convolution.
[0121] (3) The number of parameters in this method is approximately 6.569 M. At resolutions of 1024×1024, 2048×2048, and 4096×4096, the inference time for a single image is approximately 0.046 seconds, 0.235 seconds, and 1.084 seconds, respectively, with corresponding computational loads of approximately 75.88 G, 303.5 G, and 1.214 KG MACs. At a resolution of 2048×2048, the PSNR of the one-stage scheme is 39.986 dB, the SSIM is 0.9606, the computational load is 303.5 G, and the inference time is 0.235 seconds. Although the two-stage scheme using an additional sRGB chromaticity denoising network achieves a PSNR of 40.020 dB and an SSIM of 0.9610, the computational load is 537.3 G and the inference time is 0.370 seconds, indicating that the one-stage perception adaptation scheme is more suitable for practical deployment.
[0122] The above embodiments are merely typical implementations of the present invention and are not intended to limit the present invention. Any equivalent substitutions or improvements made within the scope of the claims of the present invention are within the protection scope of the present invention.
Claims
1. A perceptually friendly and noise-reducing intensity-controllable joint denoising and de-mosaic method for RAW images, characterized in that, include: First training phase: Obtain clean sRGB images, and obtain clean RAW image R*, degraded RAW image R, and clean linear RGB image through reverse processing of a differentiable bidirectional image signal processing pipeline and degradation simulation. And noise level diagram M; Perform pixel inversion on the degraded RAW image R and the noise level map M to obtain a four-channel image. and Sampling denoising control factor And branch random variable s, using the denoising control factor Construct input conditions with a mismatch between noise level and denoising intensity, and training targets containing a specified proportion of noise residuals. Combine this with supervision from regular clean targets to train a controllable joint denoising and desacrifice network. ; Second training phase: Freeze the controllable joint denoising and de-mosaicing network obtained in the first training phase and use it as the teacher network. ; A perceptual training objective is constructed using the luminance channel output by the teacher network and the chroma channel of a clean image; the student network is initialized and frozen using the weights of the teacher network. Training the perceptual adapter connected to the student network. ; Inference phase: Acquire the real degraded RAW image and its noise level map, and receive the denoising control factor set by the user or the system. The degraded RAW image and the processed image A scaled noise level map is input to the trained student network and the perception adapter, and the output is a restored linear RGB image.
2. The method according to claim 1, characterized in that, In the first training phase, the input condition where the constructed noise level does not match the denoising intensity and the training objective containing a specified proportion of noise residuals include: when At that time, The noise condition input is used as the noise condition input to the controllable joint denoising and demosaicing network, and the noise residual of the degraded RAW image relative to the clean RAW image is used. proportionally Inject a clean, linear RGB target at the Bayer sampling region; the training target is represented as: and Minimize the differences between them; when At that time, an unscaled noise level map was used. and clean linear RGB images Perform standard pairwise supervision; the training objective is represented as: and Minimize the differences between them; in, This represents the mosaic operator that projects a three-channel image onto the Bayer sampling location.
3. The method according to claim 1, characterized in that, In the second training phase, the construction of the perceptual training target using the luminance channel output by the teacher network and the chroma channel of a clean image includes: The teacher network The output is processed through the forward image signal processing flow. Mapped to the sRGB domain, then converted to the YUV domain, the teacher's luminance channel is obtained. Teacher color channel , ; Clean sRGB image Convert to the YUV domain to obtain , , ; Using the teacher brightness channel The chroma channels of the clean image , splicing together to form a perceptual training objective .
4. The method according to claim 1, characterized in that, The sensing adapter The student network encoder and the cropped copy of the intermediate block, whose inputs include , and Output control features The control features are injected into the corresponding layer of the student network via zero-initialization convolution, so that the student output is consistent with the teacher output at the start of training.
5. The method according to claim 1, characterized in that, The controllable joint denoising and desacrifice network adopts a U-shaped multi-scale encoder-intermediate layer-decoder structure, in which the encoder incorporates a degradation-aware convolutional layer (DAConv); the degradation-aware convolution extracts noise priors from the noise level map. With degenerative characteristics After concatenation along the channel dimension, spatial adaptive weights are generated sequentially through 1×1 pointwise convolution, depthwise convolution, and 1×1 pointwise convolution. The weights are rearranged into grouped convolutional kernels, and dynamic convolution is performed on the neighborhood centered at each position, outputting the result. Add the residuals together.
6. The method according to claim 5, characterized in that, The spatial adaptive weights The generation method is represented as follows: ; Wherein, Concat represents concatenation along the channel dimension, PConv represents 1×1 pointwise convolution, and DConv7 represents 7×7 depthwise convolution.
7. The method according to claim 1, characterized in that, The differentiable bidirectional image signal processing pipeline includes at least one reversible process among mosaic / demosaic, exposure compensation / white balance, color correction, tone mapping, and gamma correction; the degradation simulation includes injecting shot noise and readout noise into a clean RAW image, and the noise level map... Represented as: ; in, Indicates the total gain. This represents the variance of the readout noise.
8. The method according to claim 1, characterized in that, The first training phase uses hard logarithmic loss. Optimize the controllable joint denoising and desacrifice network: ; in, , , These represent the number of channels, height, and width, respectively, and ε is a stability constant.
9. A sensor-friendly and noise reduction intensity-controllable RAW image joint denoising and de-mosaicing device, characterized in that, include: The data construction module is used to generate clean RAW images, degraded RAW images, clean linear RGB images, and noise level maps through differentiable bidirectional image signal processing and degradation simulation. A controllable training module is used to scale the noise level map according to the denoising control factor and switch between the Bayer domain noise residual target and the three-channel clean target to train a controllable joint denoising and desacrifice network. The perception adaptation module is used to generate perception targets using the teacher's luminance channel and clean chrominance channel, and to train the perception adapter connected to the frozen student network. The degradation-aware restoration module is used to generate spatially adaptive convolution kernels based on noise priors and local content, and jointly perform RAW image denoising and de-mosaicing. The intensity control and output module receives control factors, inputs the degraded RAW image and the scaled noise level map into the trained network, and outputs an sRGB image after image signal processing.
10. An electronic device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 8.
