A noise adaptive video denoising method based on semi-blind estimation

Abstract

The application discloses a noise adaptive video denoising method based on semi-blind estimation. The method comprises the following steps: (1) constructing a semi-blind noise analysis neural network module based on a physical noise model; (2) constructing a video dataset containing heterodistribution noise, and pre-training the semi-blind noise analysis neural network module; (3) constructing a cyclic time large-span video denoising neural network module; (4) cascading the semi-blind noise analysis neural network module and the video denoising neural network module, and performing end-to-end training on the cascaded module using the video dataset containing heterodistribution noise. The application is based on the semi-blind noise analysis module, and the distribution parameter analysis and estimation of the heterodistribution noise are performed, and the distribution parameter is used to guide the video denoising module to process different camera heterodistribution noise, so that the adaptability and generalization of the video denoising algorithm in different cameras are fully improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and more particularly to a noise adaptive video denoising method based on semi-blind estimation. Background Technology

[0002] Shooting video in low-light environments presents significant challenges. Video noise is extremely noticeable in low light, severely interfering with the original video signal and making it difficult to extract the necessary scene information. Furthermore, the noise distribution is extremely complex, containing various components with different properties and behaviors. These include shot noise caused by random fluctuations in photon arrival and photoelectric conversion, leakage current from crystal material surface defects and dark current caused by carrier thermal diffusion, readout noise generated in the readout circuit, and line noise caused by circuit fluctuations and asynchronous excitation. Especially under extreme lighting conditions, this random interference cannot be described using simple probability distributions.

[0003] For different cameras, due to specific circuit designs or parameter settings, although the various noise components generally follow the same distribution, the values ​​of the distribution parameters vary significantly. Therefore, the overall noise exhibits heterogeneous distribution characteristics as the camera or camera parameter settings change, with large variations in distribution. Existing neural network algorithms can only process noisy images from specific cameras or with specific distributions. Their main steps involve first calibrating the noise distribution parameters of images from a specific camera based on a physical noise model, and then using the calibrated parameters to synthesize targeted training data to train the denoising network. This type of method involves a complex and cumbersome calibration process, and for processing noisy videos or images from different cameras, recalibration and retraining are required, resulting in a significant waste of time and computational resources. Summary of the Invention

[0004] To address the limitations of existing technologies, the present invention aims to propose an adaptive video denoising method for heterogeneous noise based on semi-blind estimation. By automatically analyzing and estimating the noise distribution parameters, a single network can remove video noise with different distributions from different cameras.

[0005] The technical solution adopted in this invention is as follows:

[0006] A noise-adaptive video denoising method based on semi-blind estimation is proposed. This method first performs distribution parameter analysis and estimation on heterogeneous noise, and then uses the distribution parameters to guide the video denoising neural network to process heterogeneous noise from different cameras.

[0007] Furthermore, the method includes the following steps:

[0008] Step 1: Construct a semi-blind noise analysis neural network module based on a physical noise model;

[0009] Step 2: Construct a video dataset containing heterogeneous noise and pre-train the semi-blind noise analysis neural network module;

[0010] Step 3: Construct a recurrent long-span video denoising neural network module, which processes video sequences with two frame rates (fast and slow) through a dual-branch approach to capture temporal features at different scales in the video.

[0011] Step 4: Cascade the semi-blind noise analysis neural network module with the video denoising neural network module, and use the video dataset containing heterogeneous noise constructed in Step 2 to perform end-to-end training on the cascaded module.

[0012] Furthermore, in step 1, a physical noise model is used to model the actual camera noise:

[0013] y i =T(K c ·β c ·(S i +D i +R i c∈{r, g, b}

[0014] Among them, y i Let K represent the pixel value at image position i, T represent the truncation operation, and K represent the pixel value at image position i. c Indicates global gain. This represents fringe noise that follows a Gaussian distribution with variance . This represents shot noise that follows a Poisson distribution, with variance and mean N. e , This represents dark current noise that follows a Poisson distribution, with variance and mean N. d , This represents the readout noise that follows a Gaussian distribution, with variance .

[0015] A semi-blind noise analysis neural network module, including several long short-term memory modules, is constructed to estimate eight noise model parameters for a given noisy video by combining spatiotemporal information.

[0016] Furthermore, in step 3, the video sequence V to be denoised is... F ={I1, I2, I3, I4, ...} is downsampled at time intervals d to obtain a low frame rate video sequence V. S ={I1, I 1+d I 1+2d I 1+3d ,…}, where F represents fast frame rate, I represents frame, and S represents low frame rate;

[0017] Constructing a cyclic fast-path encoder E based on a long short-term memory module (LSTM)F Slow-path encoder E S To extract temporal features at different scales and fuse them with the decoder D SF To reconstruct clear video frames; to convert the video sequence V F and V S The parameters are combined with the noise model parameters separately and then fed into the encoder E. F and E S Extracting features at different time scales and When time t∈{1,1+d,1+2d,1+3d,…}, the features and The mixture is then fused and fed into the fusion decoder D. SF The denoising result y is obtained t When time At that time, the features Input into fusion decoder D SF The denoising result y is obtained t .

[0018] Further, in step 4, the noise parameters estimated by the semi-blind noise analysis neural network module are spatially copied to obtain a feature map, and the feature map is merged with the noise image channel to obtain a fused feature map. The fused feature map is then sent to the video denoising neural network module for processing, and finally the denoising result is obtained.

[0019] The physical noise model employed in this invention includes modeling of truncation effects, RGB channel inconsistencies, stripe noise, shot noise, dark current, and readout noise. It accurately describes the properties of camera noise under low light or underexposure conditions. By constructing a semi-blind noise analysis neural network module, it analyzes the noise composition of a given camera video or image, obtaining a series of noise distribution parameters. These parameters then guide the video denoising neural network module for adaptive denoising of different camera noises. Compared to existing solutions, the semi-blind noise analysis neural network module constructed in this invention has the advantages of incorporating prior knowledge of the physical sources and distribution characteristics of noise, thus resulting in more accurate noise analysis; secondly, it eliminates the complex process of traditional manual calibration, requiring only the input image to be denoised for automatic noise analysis; and finally, the module's output can serve as a guide for the denoising module, helping to remove heterogeneous noise distributions from different cameras, achieving the goal of a fast single network applicable to multiple cameras. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the process of the present invention;

[0021] Figure 2 This is a schematic diagram of a semi-blind noise analysis neural network module;

[0022] Figure 3 A schematic diagram of a cyclic, long-span video denoising neural network module;

[0023] Figure 4 This is a schematic diagram of (a) the encoder and (b) the decoder in a video denoising neural network. Detailed Implementation

[0024] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0025] This embodiment provides a noise adaptive video denoising method based on semi-blind estimation, such as... Figure 1 As shown, a semi-blind noise analysis module is used to analyze and estimate the distribution parameters of heterogeneous noise. These distribution parameters then guide the video denoising module in processing heterogeneous noise from different cameras, significantly improving the adaptability and generalization of the video denoising algorithm across various cameras. The method specifically includes the following steps:

[0026] Step 1: Construct a semi-blind noise analysis neural network module based on a physical noise model;

[0027] A physical noise model is used to model real camera noise:

[0028] y i =T(K c ·β c ·(S i +D i +R i c∈{r, g, b}

[0029] y i Let K represent the pixel value at image position i, T represent the truncation operation, and K represent the pixel value at image position i. c Indicates global gain. This represents fringe noise that follows a Gaussian distribution with variance . This represents shot noise that follows a Poisson distribution, with variance and mean N. e , This represents dark current noise that follows a Poisson distribution, with variance and mean N. d , This represents the readout noise that follows a Gaussian distribution, with variance . There are 8 noise model parameters here. They reflect the distribution characteristics of noise from different cameras.

[0030] A semi-blind noise analysis neural network module containing several Long Short-Term Memory (LSTM) modules is constructed to estimate eight noise model parameters for a given noisy video. This embodiment uses, as follows: Figure 2The network structure shown includes four convolutional + activation layers, three LSTM units, and one global pooling layer. The middle two convolutional layers have a stride of 2 to achieve spatial downsampling and obtain global spatial noise information. The other two convolutional layers have a stride of 1. All convolutional kernels have a size of 3, and the last convolutional layer has 8 channels, corresponding to 8 estimated noise parameters. The remaining convolutional or LSTM units have 32 channels. The LSTM units are used to acquire information from multiple frames to help analyze the statistical characteristics of the noise, and global pooling is used to map a spatially inconsistent feature channel to a single parameter value.

[0031] Step 2: Construct a video dataset containing heterogeneous noise and pre-train the noise analysis module;

[0032] The video dataset can be in the format of 24 frames per scene, collecting no fewer than 2000 scenes in total. Based on the basic camera parameters of several common cameras such as Cano 5D Mark III, Grasshopper 3 GS3-U3-32S4C, and ASI294 Pro, the noise parameter space is established: global gain parameter K. c | c∈{r,g,b} ∈[1,50], stripe noise parameters Dark current parameter N d ∈[0.002,4], read out the noise parameters Noisy videos are generated by randomly sampling noise parameter sets from this parameter space. The same set of parameters is used for all 24 frames of a scene, as a video generally has a similar noise distribution. The generated noisy videos and noise parameters are used as input and labels, respectively, for training a noise analysis network. The L1 loss function is used. The optimization process consists of two steps: first, the pre-trained network parameters of the noise analysis module are fixed, and the network parameters of the video denoising module are optimized; second, the parameters of each network are fine-tuned.

[0033] Step 3: Construct a recurrent, long-span video denoising neural network module, such as... Figure 3 As shown, by processing video sequences at both fast and slow frame rates through dual-branch processing, the temporal features at different scales in the video are captured to achieve a large time span, thereby obtaining more video correlations and better helping to reconstruct clear videos.

[0034] The video sequence V to be denoised F = {I1,I2,I3,I4,…}, where F represents the high frame rate and I represents the frame. Time downsampling is performed at time intervals d to obtain the low frame rate video sequence V. S ={I1,I 1+d ,I 1+2d ,I 1+3d,…}, where S represents a low frame rate, and a cyclic fast-path encoder E based on a long short-term memory module LSTM is constructed. F Slow-path encoder E S To extract temporal features at different scales, and to fuse the decoder D SF To reconstruct clear video frames. V F V S The parameters are combined with the noise model parameters separately and then fed into the encoder E. F E S Extract features at different time scales When time t∈{1,1+d,1+2d,1+3d,…}, and The mixture is fused using a single convolutional layer and then fed into D. SF The denoising result y is obtained t When time At that time, Send to D SF The denoising result y is obtained t ;

[0035] Cyclic fast-path encoder E F Cyclic slow-path encoder E S , and loop fusion decoder D SF The structure adopts, for example Figure 4 The network shown, E F or E S It contains 3 convolutional layers + activation layers and 3 LSTM units. The first convolutional layer has a stride of 1, and the stride of the next two convolutional layers is set to 2 to achieve downsampling. All convolutional kernel sizes are set to 3, and the initial number of channels is set to 48, with the number of channels doubling after each downsampling layer. SF It contains 2 deconvolution + activation layers, 1 convolution + activation layer, and 2 LSTM units. The deconvolution stride is set to 2 to achieve upsampling, the convolution stride is 1, all convolution kernel sizes are set to 3, the number of channels is initially set to 48*4, and the number of channels is halved after each downsampling layer. The last convolution layer outputs a three-channel reconstructed image.

[0036] Step 4: Cascade the noise analysis module and the video denoising module, and train them end-to-end using a video dataset containing heterogeneous noise.

[0037] Noise image I t The noise parameter P is input into the noise analysis module to obtain an 8×1×1 noise parameter. The noise parameter P is then spatially copied to obtain an 8×H×W feature map P. 2d and P 2d With 3×H×W noisy image I t Channel merging yields an 11×H×W fused feature map U. t Then Ut The video is fed into the video denoising module for processing, and the denoised result y is finally obtained. t .

Claims

1. A noise adaptive video denoising method based on semi-blind estimation, characterized in that, The method first analyzes and estimates the distribution parameters of the heterogeneous distribution noise, and uses the distribution parameters to guide the video denoising neural network to process the heterogeneous distribution noise of different cameras; Specifically, the method comprises the following steps: Step 1, constructing a semi-blind noise analysis neural network module based on a physical noise model; Step 2, constructing a video dataset containing heterogeneous distribution noise, and pre-training the semi-blind noise analysis neural network module; Step 3, constructing a recurrent time large-span video denoising neural network module, which processes video sequences of fast and slow frame rates through double branches to capture different scale time features in the video; Specifically: The video sequence V to be denoised F ={I1,I2,I3,I4,…} is downsampled at time intervals d to obtain a low frame rate video sequence V. S ={I1,I 1+d ,I 1+2d ,I 1+3d ,…}, where F represents fast frame rate, I represents frame, and S represents low frame rate; Constructing a recurrent expressway encoder E based on a long short-term memory module LSTM F , a slow road encoder E S to extract time features of different scales and fuse a decoder D SF to reconstruct clear video frames; a video sequence V F and V S are respectively merged with noise model parameters, and are respectively sent into an encoder E F and E S to extract features of different time scales and When time t∈{1,1+d,1+2d,1+3d,…}, features and are fused and sent into a fusion decoder D SF to obtain a denoising result y t When time , features are sent into a fusion decoder D SF to obtain a denoising result y t ; Step 4, cascading the semi-blind noise analysis neural network module and the video denoising neural network module, and using the video dataset containing heterogeneous distribution noise constructed in step 2 to train the cascaded module end-to-end.

2. The noise adaptive video de-noising method based on semi-blind estimation according to claim 1, characterized in that, In step 1, a physical noise model is used to model the real camera noise: y i = T(K c · β c · (S i + D i + R i )), c e {r, g, b} where y i denotes the pixel value at image location i, T denotes the truncation operation, K c denotes the global gain, denotes the readout noise following a Gaussian distribution with variance S i ~ P(N e ) denotes the shot noise following a Poisson distribution with variance and mean N e , D i ~ P(N d ) denotes the dark current noise following a Poisson distribution with variance and mean N d , denotes the readout noise following a Gaussian distribution with variance A semi-blind noise analysis neural network module including several long short-term memory modules is constructed, and eight noise model parameters are estimated by combining the spatial and temporal information for a given noisy video wherein K r , K g , K b respectively represent the global gain of the red channel, the global gain of the green channel, and the global gain of the blue channel, respectively represent the stripe noise variance of the red channel, the stripe noise variance of the green channel, and the stripe noise variance of the blue channel.

3. The noise adaptive video de-noising method based on semi-blind estimation according to claim 1, characterized in that, In step 4, the noise parameters estimated by the semi-blind noise analysis neural network module are spatially replicated to obtain a feature map, and the feature map is combined with a noise image channel to obtain a fused feature map, which is then input into the video denoising neural network module for processing, and finally the denoising result is obtained.

Images