A video denoising device

Abstract

This invention provides a video denoising device, including a noise intensity estimation module, a denoising network structure, an unsupervised training module, and a parameter adjustment module. The noise intensity estimation module acquires multiple frames of noisy video and original image data, and obtains the noise intensity corresponding to the noisy video image. The unsupervised training module provides a training loss function. The denoising network structure generates multi-frame noise intensity maps by decoupling spatial domain transformation and temporal synthesis of the noise intensity, and simultaneously trains the denoising network structure according to the training loss function. The parameter adjustment module controls the temporal synthesis intensity of the denoising network structure, enabling the training of the video denoising model with only real noisy video and no noise-free video, and is easily deployed on embedded devices, supporting flexible parameter adjustment by users. The noise estimation in the noise estimation module and the unsupervised learning in the unsupervised learning module are both carried out using real noisy video, making it more practical.

Description

Technical Field

[0001] This invention relates to the field of video noise reduction, and in particular to a video noise reduction device. Background Technology

[0002] In recent years, with the rise of artificial intelligence technology, using deep learning methods for image and video denoising has become a trend. Deep learning methods are significantly better than traditional methods at denoising images and videos. However, the widespread application of deep learning-based video denoising networks in industry is still some distance away, and these networks mainly suffer from the following problems:

[0003] 1) Training data issues: Currently, the noise in the training data used by the academic community to train video denoising networks is usually artificially synthesized. However, the noise of actual image sensors is much more complex than artificially synthesized noise. Furthermore, it is very difficult to actually collect noisy and noiseless training data pairs from the same source using image sensors, making it difficult to ensure that the training data pairs are perfectly aligned.

[0004] 2) Computational complexity: Currently, some mainstream video noise reduction networks require hundreds of gigabit-level floating-point operations on a 1080P screen, requiring terabit-level or higher computing power to achieve real-time video processing in the camera's chip.

[0005] 3) Parameter adjustability issues: The neural network model trained by deep learning methods is usually a black box, and the specific meaning of the parameters is unclear. The parameters are fixed immediately after training. Therefore, it is usually difficult to adapt to different scenarios and noise by adjusting some parameters. Adjusting some parameters will limit the flexibility of product application.

[0006] To address problem 1), the academic community has developed some unsupervised learning methods, which achieve unsupervised learning by constraining the denoising results of different noisy images from the same source to be consistent. However, this unsupervised learning method is usually only suitable for training data with artificially added noise. For real image sensor images, due to factors such as unstable ambient light, camera shake, and moving objects in the scene, it is difficult to obtain videos with the same data source but different noise levels. To address problem 2), namely reducing the computational cost of video denoising networks, some lightweight networks have been proposed. However, compared with models with higher computational costs, these lightweight models usually result in a significant performance drop. To address problem 3), namely improving the flexibility of the model and adapting to noise of different intensities, current mainstream denoising networks incorporate a noise intensity map (i.e., the standard deviation of noise) as part of the network input. However, this noise intensity map is mostly a global coarse estimate and does not achieve local adaptation, so its flexibility is still insufficient. Summary of the Invention

[0007] The purpose of this invention is to provide a video denoising device that can perform unsupervised training and flexibly adjust parameters.

[0008] To address the aforementioned problems, this invention provides a video denoising device, comprising a noise intensity estimation module, a denoising network structure, an unsupervised training module, and a parameter adjustment module.

[0009] The noise intensity estimation module collects multiple frames of noisy video and raw image data, and obtains the noise intensity corresponding to each frame of the noisy video based on the multiple frames of noisy video and raw image data.

[0010] The unsupervised training module provides a training loss function to the denoising network structure;

[0011] The denoising network structure takes into account the noise intensities corresponding to multiple frames of noisy video and multiple frames of noisy video images, and generates multi-frame noise intensity maps by decoupling spatial domain transformation and temporal synthesis on the noise intensities of the multiple noisy video frames and the noise intensities corresponding to each frame of the noisy video image. Simultaneously, the denoising network structure is trained according to the training loss function to obtain a trained neural network model; and

[0012] The parameter adjustment module is used to control the temporal synthesis intensity of the denoising network structure by fine-tuning the noise intensity map, so as to output a finely tuned trained neural network model.

[0013] Optionally, the raw image data includes system noise and Poisson noise.

[0014] Furthermore, the noise intensity estimation module is used to adjust different image sensor gains under the condition that no light signal enters the image sensor, record multiple frames of noisy video at each gain, and calculate the standard deviation of the noisy video at each gain, and use the standard deviation as the standard deviation of the system noise at that gain.

[0015] Furthermore, the relationship between the standard deviation and the gain satisfies the following formula:

[0016] (1)

[0017] in, denoted as the standard deviation of the system noise; Gain is the gain of the image sensor; k and b are both fitting parameters.

[0018] Furthermore, the Poisson noise intensity satisfies the following formula:

[0019] (2)

[0020] (3)

[0021] (4)

[0022] (5)

[0023] in, The standard deviation of Poisson noise; The mean of Poisson noise; QE is the quantum efficiency of converting a photon into an electron; The maximum capacity to collect electrons at each pixel location of the image sensor; The maximum value of the digital signal; The value is a noisy digital signal. The standard deviation of the system noise is given by ; Gain is the gain of the image sensor; Filter represents a side-preserving filter, and is used in digital signal processing. and noise standard deviation For input.

[0024] Furthermore, the noise standard deviation at each pixel location of each of the noisy video images satisfies the following formula:

[0025] (6)

[0026] in, The standard deviation of the system noise; The standard deviation of Poisson noise; The noise standard deviation for each pixel.

[0027] Optionally, the denoising network structure includes multiple denoising blocks connected in series. Each denoising block receives multiple frames of the noisy video and the noise intensity corresponding to each frame of the noisy video image, and obtains a noise intensity map based on each frame of the noisy video image and the noise intensity corresponding to each frame of the noisy video image.

[0028] Furthermore, the denoising block receives the noisy video and the noise intensity corresponding to each frame of the noisy video image. First, it performs spatial domain transformation on each frame of the noisy video image, then performs spatial domain transformation on the noise intensity corresponding to each frame of the noisy video image, and then generates synthesis coefficients based on the transformed noise intensity. Then, it generates synthesized multi-frame noisy video images and the noise intensity corresponding to each frame of the synthesized noisy video image based on the synthesis coefficients, thereby obtaining multi-frame noisy video images and a noise intensity map corresponding to the multi-frame noisy video images.

[0029] Furthermore, each frame of the noisy video image after spatial transformation satisfies the following formula:

[0030] (7)

[0031] in, The k-th frame is a noisy video image; is for convolution operation; W is the convolution kernel used to perform spatial transformation on noisy video images.

[0032] Furthermore, the noise intensity corresponding to each frame of the noisy video image after spatial transformation satisfies the following formula:

[0033] (8)

[0034] in, To perform a square operation on each pixel position of the noise intensity map corresponding to each frame of the noisy video image; This is to perform pixel-by-pixel squaring on the convolution kernel.

[0035] Furthermore, the synthesis coefficients for generating multi-frame noisy video images based on the transformed noise intensity map satisfy the following formula:

[0036] (9)

[0037] (10)

[0038] (11)

[0039] in, Noise intensity map after spatial transformation of n frames Tensors formed by splicing; The kernel is the convolution kernel; the division and summation in formula (9) are pixel-by-pixel position operations; is the synthesis coefficient.

[0040] Furthermore, the synthesized k-th noisy video image and the synthesized k-th frame noise intensity image They are respectively:

[0041] (12)

[0042] (13)

[0043] in, This refers to the pixel value in the y-th row, x-th column, and l-th channel of the synthesized noisy video image; This represents the pixel value in the y-th row, x-th column, and l-th channel of the synthesized noisy video noise standard deviation image.

[0044] Furthermore, the unsupervised training module includes a training loss function, such that each denoising block can be configured with a training loss function.

[0045] Furthermore, the training loss function satisfies the following formula:

[0046] (14)

[0047] in, This represents the L2 norm of the synthesized noise intensity. The L2 norm of the difference between the output frame and the input frame; To control the proportion of the two objectives in the loss function.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] This invention provides a video denoising device, including a noise intensity estimation module, a denoising network structure, an unsupervised training module, and a parameter adjustment module. The noise intensity estimation module acquires multiple frames of noisy video and original image data, and obtains the noise intensity corresponding to each frame of the noisy video based on the multiple frames of noisy video and the original image data. The unsupervised training module provides a training loss function to the denoising network structure. The denoising network structure inputs the noise intensity corresponding to the multiple frames of noisy video and the multiple frames of noisy video images, and generates a multi-frame noise intensity map by decoupling spatial domain transformation and temporal synthesis on the multiple frames of noisy video and the noise intensity corresponding to each frame of the noisy video image. Simultaneously, the denoising network... The network structure is trained according to the training loss function to obtain a trained neural network model; and the parameter adjustment module is used to control the temporal synthesis intensity of the denoising network structure by fine-tuning the noise intensity map to output a finely tuned trained neural network model. This allows for training of video denoising models with only real noisy videos and no videos without noise sources, and it is easy to deploy on embedded devices, supporting flexible parameter adjustment by users. The noise estimation module and the unsupervised learning module of this invention are both designed for real noisy videos in industry, which is more practical than the artificially synthesized noise training used in current academia. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of the structure of a video denoising device according to an embodiment of the present invention;

[0051] Figure 2 This is a schematic diagram illustrating spatial transformation of each frame of noisy video image according to an embodiment of the present invention.

[0052] Figure 3 This is a schematic diagram of spatial transformation of the noise intensity corresponding to each frame of noisy video image according to an embodiment of the present invention.

[0053] Figure 4This is a schematic diagram of the denoising network structure 2 provided in an embodiment of the present invention.

[0054] Explanation of reference numerals in the attached figures:

[0055] 1-Noise intensity estimation module; 2-Denoising network structure; 3-Unsupervised training module; 4-Parameter tuning module. Detailed Implementation

[0056] The following will provide a further detailed description of a video denoising apparatus according to the present invention. The invention will now be described in more detail with reference to the accompanying drawings, which illustrate preferred embodiments of the invention. It should be understood that those skilled in the art can modify the invention described herein while still achieving its advantageous effects. Therefore, the following description should be understood as being of general knowledge to those skilled in the art and is not intended to limit the invention.

[0057] For clarity, not all features of the actual embodiments are described. In the following description, well-known functions and structures are not detailed in detail, as they would obscure the invention with unnecessary detail. It should be understood that in the development of any actual embodiment, numerous implementation details must be made to achieve the developer's specific objectives, such as changes from one embodiment to another according to limitations related to the system or business. Furthermore, it should be understood that such development work may be complex and time-consuming, but is merely routine work for those skilled in the art.

[0058] To make the objectives and features of the present invention more apparent and understandable, the specific embodiments of the present invention will be further described below with reference to the accompanying drawings. It should be noted that the drawings are all in a very simplified form and use non-precise ratios, and are only used to conveniently and clearly assist in illustrating the objectives of the embodiments of the present invention.

[0059] This embodiment provides a video denoising device comprising a noise intensity estimation module 1, a denoising network structure 2, an unsupervised training module 3, and a parameter adjustment module 4. The noise intensity estimation module 1 acquires multiple frames of noisy video and original image data, and obtains the noise intensity corresponding to each frame of the noisy video based on the multiple frames of noisy video and the original image data. The denoising network structure 2 takes the multiple frames of noisy video and their corresponding noise intensities as input, and generates a multi-frame noise intensity map by decoupling spatial domain transformation and temporal synthesis of the multiple frames of noisy video and their corresponding noise intensities. The unsupervised training module 3 provides a training loss function to the denoising network structure 2, and the denoising network structure 2 is trained according to the training loss function to obtain a trained neural network model. The parameter adjustment module 4 controls the temporal synthesis intensity of the denoising network structure 2 by fine-tuning the noise intensity map to output a finely tuned trained neural network model.

[0060] The noise intensity estimation module 1 can estimate the noise intensity at each pixel position based on hardware prior information to obtain the noise standard deviation at each pixel position of each frame of noisy video image in the noisy video.

[0061] The noise intensity estimation module 1 can acquire raw image data and multi-frame noisy video from the image sensor at different gains, and perform noise intensity estimation on the raw image data and multi-frame noisy video. The multi-frame noisy video includes multi-frame noisy video images, and the raw image data includes system noise and Poisson noise. The module can also estimate the noise intensity of the system noise and Poisson noise respectively.

[0062] In detail, the noise intensity estimation module 1 can adjust different image sensor gains under the condition that no light signal enters the image sensor, record multiple frames of noisy video at each gain, and calculate the standard deviation of the noisy video at each gain, and use the standard deviation as the standard deviation of the system noise at that gain.

[0063] The relationship between standard deviation and gain satisfies the following formula:

[0064] (1)

[0065] in, denoted as the standard deviation of the system noise; Gain is the gain of the image sensor, which is the product of the analog gain and the digital gain; k and b are both fitting parameters.

[0066] Since Poisson noise originates from optical signals and is based on the physical processes of photon-to-electron and analog-to-digital signal conversion, the following formula can be established to determine the relationship between Poisson noise intensity and noise mean:

[0067] (2)

[0068] (3)

[0069] (4)

[0070] (5)

[0071] in, The standard deviation of Poisson noise; The mean of Poisson noise; QE is the quantum efficiency of converting photons into electrons, which is a hardware parameter of image sensors, and the quantum efficiency of the RGB three channels is usually different; The maximum capacity to collect electrons at each pixel location of an image sensor is also a hardware parameter. This represents the maximum value of a digital signal; for example, the maximum value of a 12-bit digital signal is 4095. The standard deviation of noise; The value is a noisy digital signal. The standard deviation of the system noise is given by ; Gain is the gain of the image sensor; Filter represents a side-preserving filter, and is used in digital signal processing. and noise standard deviation For input.

[0072] Based on the assumption that the mean of system noise is 0, as long as the Poisson noise is estimated... The mean of the signal is obtained, and the standard deviation of the Poisson noise is calculated using formulas (1) and (2). The digital signal at each pixel position in space (i.e., Poisson noise) is obtained. The mean can be estimated using bilateral filtering, as shown in Equation (5); bilateral filtering requires a prior noise standard deviation as input, given by Equations (3) and (4).

[0073] After estimating the standard deviation of the system noise and the standard deviation of the Poisson noise, the noise standard deviation at each pixel location is obtained, thus obtaining the noise intensity map of the denoising block of the input denoising network structure 2.

[0074] The noise standard deviation at each pixel location satisfies the following formula:

[0075] (6)

[0076] in, The standard deviation of the system noise; The standard deviation of Poisson noise; The noise standard deviation for each pixel.

[0077] It should be noted that the noise estimation principle here is mainly based on the noisy video images acquired by the image sensor. This estimation method is also applicable to YUV or sRGB data. YUV or sRGB data are usually obtained by performing some linear and nonlinear transformations on the noisy video images. By performing corresponding linear and nonlinear transformations on the noise intensity of the noisey video images estimated here, the noise intensity estimate on the YUV or sRGB data can be obtained.

[0078] Because current mainstream global noise estimation methods provide coarse estimates without addressing location differences, the resolution of the denoised image is reduced. However, the noise intensity estimation module 1 in this embodiment can provide pixel-by-pixel noise intensity estimates based on prior physical information from the image sensor. This estimation method has high reliability and strong adaptability, providing a reliable basis for temporal synthesis of pixel-by-pixel positions in the subsequent noise reduction process.

[0079] The denoising network structure 2 uses a spatial transformation method similar to that used for noise intensity and noisy video images to track the image denoising process and reflect the degree of noise reduction in noisy video images. At the same time, it uses the transformed noise intensity to generate new temporal synthesis coefficients and then performs noisy video image synthesis to further reduce the noise in noisy video images.

[0080] The denoising network structure 2 includes multiple denoise blocks connected in series. Each denoise block receives multiple frames of noisy video collected by the noise intensity estimation module 1 and the noise intensity corresponding to each frame of noisy video image. Based on each frame of noisy video image and the noise intensity corresponding to each frame of noisy video image, a noise intensity map is obtained. It can be seen that the denoising network structure 2 has a simple structure, low computational load, and is easy to deploy on embedded devices.

[0081] Specifically, the denoising block receives the noisy video and the noise intensity corresponding to each frame of the noisy video image provided by the previous denoising block, or receives multiple frames of noisy video and the noise intensity corresponding to each frame of the noisy video image provided by the noise intensity estimation module 1, and first performs spatial transformation on each frame of the noisy video image (e.g., Figure 2 As shown), then perform spatial transformation on the noise intensity corresponding to each frame of noisy video image (e.g. Figure 3 As shown), then synthesis coefficients are generated based on the transformed noise intensity; then, based on the synthesis coefficients, synthesized multi-frame noisy video images and the noise intensity corresponding to each synthesized noisy video image are generated, and multi-frame output noisy video images and noise intensity maps corresponding to multi-frame output noisy video images are obtained and output. These multi-frame output noisy video images and noise intensity maps corresponding to multi-frame output noisy video images can be used as input for the next denoising block, and the above calculation process is repeated (e.g., ...). Figure 4 (As shown).

[0082] Each frame of noisy video image after spatial transformation satisfies the following formula:

[0083] (7)

[0084] in, Let k be a noisy video image with shape . h is the height of the noisy video image in frame k; w is the width of the noisy video image in frame k; c is the number of channels of the noisy video image in frame k. For convolution operation; W is the convolution kernel used to perform spatial transformation on noisy video images, and its shape is... , Indicates the height of the convolution kernel. This indicates the width of the convolution kernel.

[0085] To ensure that the norm of the convolution kernel is 1, let ; It is shape and A consistent convolutional kernel, which is the actual training parameter; For the small quantity protected by division, it is usually taken as 1e-8; The noisy video image after spatial transformation of the k-th frame convolved with W has the same shape. .

[0086] The noise intensity of each noisy video frame after spatial transformation satisfies the following formula:

[0087] (8)

[0088] in, To perform a square operation on each pixel position of the noise intensity map corresponding to each frame of noisy video image; This is to perform pixel-by-pixel squaring on the convolution kernel.

[0089] The synthesis coefficient map of n frames of noisy video images generated based on the transformed noise intensity map satisfies the following formula:

[0090] (9)

[0091] (10)

[0092] (11)

[0093] in, Noise intensity map after spatial transformation of n frames The tensor formed by splicing has the following shape: ; The kernel is a convolution kernel, and its shape is... The division and summation in formula (9) are both pixel-by-pixel position operations; The coefficients are the synthesis coefficients, and their shape is... .

[0094] The denoising network structure 2 in this embodiment is simplified in that only formulas (7), (8) and (9) involve convolution operations. The spatial transformation of formulas (7) and (8) only applies to a single frame image. Usually, the number of channels in a single frame of noisy video image is only 3 or 4. The small number of channels results in a small amount of computation. The convolution kernel size of formula (9) is 1x1. The small convolution kernel results in a small amount of computation. All other operations are pixel-wise operations.

[0095] (12)

[0096] (13)

[0097] in, This is the synthesized k-th noisy video image; This refers to the pixel value in the y-th row, x-th column, and l-th channel of the synthesized noisy video image; This represents the pixel value in the y-th row, x-th column, and l-th channel of the synthesized noisy video noise standard deviation image.

[0098] Compared to most current deep networks where the physical meaning of intermediate computation processes is unclear, the denoising network structure 2 in this embodiment completely decouples spatial and temporal operations, making the physical meaning of each computation step clear and highly interpretable.

[0099] The unsupervised training module 3 includes a training loss function, which allows each denoising block to be set with a training loss function. This training loss function can easily achieve unsupervised learning training when only noisy videos are available and their corresponding noise-free source videos are difficult to obtain. Furthermore, this training loss function can be applied to each small computational unit in the denoising network structure 2, ensuring that the denoising network structure 2 is fully trained.

[0100] Since the purpose of synthesizing multi-frame noisy video images and their corresponding noise intensities is to minimize noise and improve the signal-to-noise ratio, while ensuring that motion information is not covered in motion areas, the training loss function can be set to satisfy the following formula:

[0101] (14)

[0102] in, The L2 norm of the synthesized noise intensity is the lowest possible signal-to-noise ratio for the synthesized noisy video image. The L2 norm of the difference between the output frame and the input frame is used to ensure that motion region information is not overwritten during compositing. To control the proportion of the two objectives in the loss function, these are adjustable hyperparameters. It is known that this training loss function does not require a clean image source as a reference, enabling unsupervised video denoising training. Furthermore, each denoising block can be assigned a training loss function, ensuring that each layer in the denoising network structure 2 (i.e., each denoising block corresponds to one layer) receives sufficient training.

[0103] In practical applications after the unsupervised training module 3 has completed training the denoising network structure 1, the parameter adjustment module 4 can control the temporal synthesis intensity of each denoising block by fine-tuning the input noise intensity map. Specifically, by multiplying each feature intensity image by some global or local scaling factors, this parameter adjustment method allows users to flexibly fine-tune a trained neural network model based on the results.

[0104] like Figure 1 As shown, the video denoising device based on this embodiment can generate a training device for a video denoising network. A set of noisy videos is taken, and the noise intensity estimation module 1 estimates the noise intensity of each frame of the noisy video image in the set of noisy videos. The noisy videos and the corresponding noise intensities of the noisy video images form the training data. Using the designed denoising network structure 2, they are input together into the unsupervised training module 3 for training and learning. After training a neural network model, adjustable parameters are installed on the neural network model for flexible use by the user.

[0105] The use of the video denoising device will be explained using the noisy original video acquired by the image sensor as an example.

[0106] First, ensuring no light signal enters the image sensor, adjust different image sensor gain values. For each gain, record 1000 frames of raw video data. Calculate the standard deviation of the 1000 frames of raw data as the standard deviation of the system noise at that gain. Record 100 sets of system noise standard deviations corresponding to 100 gains. Fit the formula for the relationship between the system noise standard deviation and the gain as follows: After the system noise intensity formula is fitted, 100 sets of noisy videos with different gain amplitudes are collected. At this time, light signals need to be present. In order to enhance the training robustness, the scenes are required to be rich, including moving scenes as well as bright and low-light scenes. Each set of noisy videos contains 100 frames of images, and the noise intensity is estimated for each frame of image. The parameters in formula (2) Equals 60000, The QE values ​​of the three RGB channels are 0.9, 0.7, and 0.6, respectively, which is equal to 4095. The Gain value of each video frame is obtained. Based on this, the prior noise intensity at each pixel position is calculated by formulas (3) and (4). The prior noise intensity is used to perform bilateral filtering with a 5x5 window to obtain a Poisson noise mean estimate at each pixel position. Then, the Poisson noise standard deviation at each pixel position is obtained by formula (2). Finally, the noise standard deviation at each pixel position is obtained by formula (6). Thus, the noise intensity map of the input denoising network structure is obtained. 100 sets of noisy videos and their corresponding noise intensities constitute the training data.

[0107] Next, taking the input of 5 frames of noisy video into the denoising network structure as an example, since each frame of raw data has four channels, taking 1080P raw data as an example, the size of the input raw data is 540x960x4. In a denoising block module, the convolution computation of spatial domain transformation is 960x540x3x3x4x4x10, which is about 712M multiply-accumulate operations. The convolution computation of obtaining temporal synthesis coefficients is 960x540x20x5, which is about 50M multiply-accumulate operations. That is to say, the convolution computation in a denoising block is about 0.74G multiply-accumulate operations. If 15 denoising blocks are connected in series to achieve the video denoising task with a receptive field size of 31x31, the entire denoising network structure has a total of about 11G multiply-accumulate operations, which is more than one order of magnitude smaller than the computation of most video denoising networks in the current academic community.

[0108] Next, model training is performed in the unsupervised learning module. A 256x256 spatial region is randomly selected from the training data, and five consecutive frames of noisy video are randomly selected from the temporal data as input data for the denoising network structure. Unsupervised training is performed using the loss function designed in formula (14). The hyperparameter in formula (14) is set to 1, and a training loss function is set for each denoising block to ensure that the denoising network structure can be fully trained.

[0109] Finally, after training the network model designed above, a global scaling factor is added to the input noise intensity image of each denoising block and multiplied by each noise intensity image, allowing the user to control the intensity of temporal synthesis in each denoising block.

[0110] In summary, the video denoising device provided by this invention can train a video denoising model when there is only real noisy video and no video without noise source. It can also be easily deployed on embedded devices and supports users to flexibly adjust parameters. The noise estimation module and the unsupervised learning module of this invention are both designed for real noisy videos in the industry. Compared with the current academic practice of training with artificially synthesized noise, it is more practical.

[0111] Furthermore, it should be noted that, unless otherwise specified or indicated, the terms "first" and "second" in the specification are used only to distinguish the various components, elements, steps, etc. in the specification, and are not used to indicate the logical or sequential relationships between the various components, elements, steps, etc.

[0112] It is understood that although the present invention has been disclosed above with reference to preferred embodiments, these embodiments are not intended to limit the present invention. For any person skilled in the art, many possible variations and modifications can be made to the technical solutions of the present invention based on the disclosed technical content, or equivalent embodiments can be modified accordingly, without departing from the scope of the present invention. Therefore, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A video denoising apparatus, characterized by comprising: It includes a noise intensity estimation module, a denoising network structure, an unsupervised training module, and a parameter tuning module. The noise intensity estimation module collects multiple frames of noisy video and raw image data, and obtains the noise intensity corresponding to each frame of the noisy video based on the multiple frames of noisy video and raw image data. The unsupervised training module provides a training loss function to the denoising network structure; The denoising network structure takes multiple frames of noisy video and multiple frames of noisy video images as input, and generates a multi-frame noise intensity map by decoupling spatial domain transformation and temporal synthesis on the multiple frames of noisy video and the noise intensity corresponding to each frame of noisy video image. At the same time, the denoising network structure is trained according to the training loss function to obtain a trained neural network model. as well as The parameter adjustment module is used to control the temporal synthesis intensity of the denoising network structure by fine-tuning the noise intensity map, so as to output a finely tuned trained neural network model.

2. The video de-noising apparatus of claim 1, wherein, The raw image data includes system noise and Poisson noise.

3. The video de-noising apparatus of claim 2, wherein, The noise intensity estimation module is used to adjust different image sensor gains under the condition that no light signal enters the image sensor. At each gain, it records multiple frames of noisy video, calculates the standard deviation of the noisy video under each gain, and uses the standard deviation as the standard deviation of the system noise under that gain.

4. The video denoising device as described in claim 3, characterized in that, The relationship between the standard deviation and the gain satisfies the following formula: （1） wherein, is the standard deviation of the system noise; Gain is the gain of the image sensor; k and b are both fitting parameters.

5. The video de-noising apparatus of claim 4, wherein, The Poisson noise intensity satisfies the following formula: （2） （3） （4） （5） in, The standard deviation of Poisson noise; The mean of Poisson noise; QE is the quantum efficiency of converting a photon into an electron; The maximum capacity to collect electrons at each pixel location of the image sensor; The maximum value of the digital signal; The value is a noisy digital signal. The standard deviation of the system noise is given by ; Gain is the gain of the image sensor; Filter represents a side-preserving filter, and is used in digital signal processing. and noise standard deviation For input.

6. The video de-noising apparatus of claim 5, wherein, The noise standard deviation of each pixel position in each of the noisy video images satisfies the following formula: （6） wherein, is the standard deviation of the system noise; is the standard deviation of the Poisson noise; is the standard deviation of the noise for each pixel.

7. The video de-noising apparatus of claim 1, wherein The denoising network structure includes multiple denoising blocks connected in series. Each denoising block receives multiple frames of the noisy video and the noise intensity corresponding to each frame of the noisy video image, and obtains a noise intensity map based on each frame of the noisy video image and the noise intensity corresponding to each frame of the noisy video image.

8. The video de-noising apparatus of claim 7, wherein, The denoising block receives the noisy video and the noise intensity corresponding to each frame of the noisy video image. First, it performs spatial domain transformation on each frame of the noisy video image, then performs spatial domain transformation on the noise intensity corresponding to each frame of the noisy video image, and then generates synthesis coefficients based on the transformed noise intensity. Then, it generates synthesized multi-frame noisy video images and the noise intensity corresponding to each frame of the synthesized noisy video image based on the synthesis coefficients, thereby obtaining multi-frame noisy video images and a noise intensity map corresponding to the multi-frame noisy video images.

9. The video denoising apparatus as described in claim 8, characterized in that, Each frame of noisy video image after spatial transformation satisfies the following formula: （7） wherein is the kth frame of the noisy video image; is a convolution operation; W is a convolution kernel used to perform spatial transformation on the noisy video image.

10. The video de-noising apparatus of claim 9, wherein, The noise intensity of each frame of the noisy video image after spatial transformation satisfies the following formula: （8） in, To perform a square operation on each pixel position of the noise intensity map corresponding to each frame of the noisy video image; This is to perform pixel-by-pixel squaring on the convolution kernel.

11. The video de-noising apparatus of claim 10, wherein, The synthesis coefficients for generating multi-frame noisy video images based on the transformed noise intensity map satisfy the following formula: （9） （10） （11） wherein, is the n-th frame spatially transformed noise intensity map is the tensor formed by concatenation; is the convolution kernel; both the division and summation in equation (9) are pixel-wise operations; is the synthesis coefficient.

12. The video de-noising apparatus of claim 10, wherein, the kth frame of the synthesized noisy video image and the kth frame of the synthesized noisy intensity image respectively: （12） （13） in, This refers to the pixel value in the y-th row, x-th column, and l-th channel of the synthesized noisy video image; This represents the pixel value in the y-th row, x-th column, and l-th channel of the synthesized noisy video noise standard deviation image.

13. The video de-noising apparatus of claim 7, wherein, The unsupervised training module includes a training loss function, such that each denoising block can be configured with a training loss function.

14. The video de-noising apparatus of claim 13, wherein, The training loss function satisfies the following formula: （14） in, This represents the L2 norm of the synthesized noise intensity. The L2 norm of the difference between the output frame and the input frame; To control the proportion of the two objectives in the loss function.

Images