Video noise reduction method, device and electronic equipment

Abstract

This application proposes a video denoising method, apparatus, and electronic device. The method includes: acquiring a video to be processed, wherein each frame of the video includes a low-frequency component and a Laplacian component; for each frame of the video, determining a low-frequency component denoising result based on the low-frequency component of the image and the low-frequency component denoising result of the previous frame corresponding to the image; determining an intermediate image corresponding to the image based on the low-frequency component denoising result and the Laplacian component; performing edge-preserving filtering on the intermediate image to obtain a denoised image corresponding to the image. Thus, by using a denoising neural network to denoise the low-frequency component in the image, and then using edge-preserving filtering to denoise the image, the denoising effect of the low-frequency and high-frequency components in the image is ensured, while reducing the computational load.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a video noise reduction method, apparatus and electronic device. Background Technology

[0002] Current video denoising methods mainly involve acquiring images from a video, inputting these images into a deep neural network, and obtaining the denoised image output by the deep neural network. However, deep neural networks are poor at denoising high-frequency components in images, resulting in unnatural denoised images. Furthermore, deep neural networks require full-resolution input images, leading to high computational costs and excessive hardware overhead for deployment. Summary of the Invention

[0003] This application aims to at least partially address one of the technical problems in the related art.

[0004] This application proposes a video denoising method that uses a denoising neural network to denoise the low-frequency components in an image, and then uses edge-preserving filtering to denoise the image, ensuring the denoising effect of both low-frequency and high-frequency components in the image, while reducing the computational load.

[0005] The first aspect of this application proposes a video denoising method, the method comprising: acquiring a video to be processed, wherein each frame of the video includes a low-frequency component and a Laplacian component; for each frame of the video, determining a low-frequency component denoising result of the image based on the low-frequency component of the image and the low-frequency component denoising result of the previous frame corresponding to the image; determining an intermediate image corresponding to the image based on the low-frequency component denoising result of the image and the Laplacian component; and performing edge-preserving filtering on the intermediate image to obtain a denoised image corresponding to the image.

[0006] The video denoising method of this application embodiment acquires a video to be processed, in which each frame of the video includes a low-frequency component and a Laplacian component. For each frame of the video, the low-frequency component denoising result of the image is determined based on the low-frequency component of the image and the low-frequency component denoising result of the previous frame of the image. Based on the low-frequency component denoising result and the Laplacian component, the corresponding intermediate image is determined. The intermediate image is then subjected to edge-preserving filtering to obtain the denoised image. Thus, by using a denoising neural network to denoise the low-frequency component in the image and then using edge-preserving filtering to denoise the image, the denoising effect of the low-frequency and high-frequency components in the image is ensured, while reducing the computational load.

[0007] Optionally, acquiring the video to be processed, wherein each frame of the video includes a low-frequency component and a Laplacian component, includes: acquiring the video to be processed, wherein the video includes multiple frames; performing k-fold downsampling on each frame of the video to obtain a low-frequency component in the image; performing k-fold upsampling on the low-frequency component to obtain a first upsampling result; and determining the Laplacian component in the image based on the image and the first upsampling result.

[0008] Optionally, determining the low-frequency component denoising result of an image for each frame of the video based on the low-frequency component of the image and the low-frequency component denoising result of the previous frame corresponding to the image includes: for each frame of the video, inputting the low-frequency component of the image and the low-frequency component denoising result of the previous frame corresponding to the image into a denoising neural network to obtain the temporal mask and spatial denoising residual output by the denoising neural network; and determining the low-frequency component denoising result of the image based on the temporal mask, the spatial denoising residual, the low-frequency component, and the low-frequency component denoising result of the previous frame.

[0009] Optionally, the training method of the denoising neural network includes: acquiring at least one original video and an initial denoising neural network, wherein the original video is acquired in a noise-free environment; adding noise to each frame of the original image in the original video according to a sampling rate of k to obtain a noisy image corresponding to the original image; and training the initial denoising neural network based on at least one original video and the noisy image corresponding to the original image in the original video to obtain a trained denoising neural network.

[0010] Optionally, determining the intermediate image corresponding to the image based on the low-frequency component noise reduction result and the Laplacian component of the image includes: performing k-fold upsampling processing on the low-frequency component noise reduction result to obtain a second upsampling result; and performing fusion processing on the second upsampling result and the Laplacian component to obtain the intermediate image corresponding to the image.

[0011] Optionally, after determining the intermediate image corresponding to the image based on the low-frequency component denoising result and the Laplacian component of the image, the method further includes: performing pre-denoising processing on the intermediate image using a Bayer luminance channel filter to obtain a pre-denoising result of the intermediate image; determining a motion region confidence mask of the intermediate image based on the pre-denoising result of the intermediate image and the pre-denoising result of the intermediate image corresponding to the previous frame image; determining a temporal denoising result of the intermediate image based on the motion region confidence mask of the intermediate image, the pre-denoising result of the intermediate image, and the temporal denoising result of the intermediate image corresponding to the previous frame image; and updating the intermediate image based on the temporal denoising result of the intermediate image.

[0012] A second aspect of this application provides a video noise reduction apparatus, comprising: an acquisition module for acquiring a video to be processed, wherein each frame of the video includes a low-frequency component and a Laplacian component; a first determination module for determining a low-frequency component noise reduction result of each frame of the video based on the low-frequency component of the image and the low-frequency component noise reduction result of the previous frame corresponding to the image; a second determination module for determining an intermediate image corresponding to the image based on the low-frequency component noise reduction result of the image and the Laplacian component; and a filtering processing module for performing edge-preserving filtering processing on the intermediate image to obtain a noise-reduced image corresponding to the image.

[0013] The video denoising apparatus of this application acquires a video to be processed, wherein each frame of the video includes a low-frequency component and a Laplacian component. For each frame of the video, a low-frequency component denoising result is determined based on the low-frequency component of the image and the low-frequency component denoising result of the previous frame. An intermediate image is determined based on the low-frequency component denoising result and the Laplacian component. The intermediate image is then subjected to edge-preserving filtering to obtain the denoised image. Thus, by using a denoising neural network to denoise the low-frequency component in the image and then using edge-preserving filtering to denoise the image, the denoising effect of both low-frequency and high-frequency components in the image is ensured, while reducing the computational load.

[0014] Optionally, the acquisition module is specifically used to: acquire a video to be processed, the video including multiple frames of images; perform k-fold downsampling on each frame of the video to obtain low-frequency components in the image; perform k-fold upsampling on the low-frequency components to obtain a first upsampling result; and determine the Laplacian component in the image based on the image and the first upsampling result.

[0015] Optionally, the first determining module is specifically used to: for each frame of the video, input the low-frequency component of the image and the low-frequency component noise reduction result of the previous frame corresponding to the image into a noise reduction neural network to obtain the temporal mask and spatial noise reduction residual output by the noise reduction neural network; and determine the low-frequency component noise reduction result of the image based on the temporal mask, the spatial noise reduction residual, the low-frequency component, and the low-frequency component noise reduction result of the previous frame.

[0016] Optionally, the first determining module is further configured to: acquire at least one original video and an initial denoising neural network, wherein the original video is acquired in a noise-free environment; add noise to each frame of the original image in the original video according to a sampling rate of k to obtain a noisy image corresponding to the original image; and train the initial denoising neural network based on at least one original video and the noisy image corresponding to the original image in the original video to obtain a trained denoising neural network.

[0017] Optionally, the second determining module is specifically used to perform k-fold upsampling on the low-frequency component noise reduction result to obtain a second upsampling result; and to perform fusion processing on the second upsampling result and the Laplacian component to obtain an intermediate image corresponding to the image.

[0018] Optionally, the device further includes: a pre-denoising module, a third determining module, a fourth determining module, and an updating module; the pre-denoising module is used to perform pre-denoising processing on the intermediate image using a Bayer luminance channel filter to obtain a pre-denoising result of the intermediate image; the third determining module is used to determine a motion region confidence mask of the intermediate image based on the pre-denoising result of the intermediate image and the pre-denoising result of the intermediate image corresponding to the previous frame image; the fourth determining module is used to determine a temporal denoising result of the intermediate image based on the motion region confidence mask of the intermediate image, the pre-denoising result of the intermediate image, and the temporal denoising result of the intermediate image corresponding to the previous frame image; the updating module is used to update the intermediate image based on the temporal denoising result of the intermediate image.

[0019] A third aspect of this application provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the video noise reduction method proposed in the first aspect of this application.

[0020] A fourth aspect of this application provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the video noise reduction method proposed in the first aspect of this application.

[0021] The fifth aspect of this application provides a computer program product that, when executed by an instruction processor, performs the video noise reduction method proposed in the first aspect of this application.

[0022] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0023] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0024] Figure 1 This is a schematic flowchart of the video noise reduction method provided in Embodiment 1 of this application;

[0025] Figure 2 This is a schematic flowchart of the video noise reduction method provided in Embodiment 2 of this application;

[0026] Figure 3 This is a diagram illustrating video noise reduction.

[0027] Figure 4 This is a schematic diagram of the video noise reduction device provided in Embodiment 3 of this application;

[0028] Figure 5 This is a block diagram illustrating an electronic device for a video noise reduction method according to an exemplary embodiment. Detailed Implementation

[0029] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0030] Current video denoising methods mainly involve acquiring images from a video, inputting these images into a deep neural network, and obtaining the denoised image output by the deep neural network. However, deep neural networks are poor at denoising high-frequency components in images, resulting in unnatural denoised images. Furthermore, deep neural networks require full-resolution input images, leading to high computational costs and excessive hardware overhead for deployment.

[0031] To address the aforementioned issues, this application proposes a video noise reduction method, apparatus, electronic device, and storage medium.

[0032] Figure 1 This is a flowchart illustrating the video noise reduction method provided in Embodiment 1 of this application. It should be noted that this video noise reduction method can be applied to a video noise reduction device. The video noise reduction device can be applied, for example, to any electronic device, or to the CPU processor of an electronic device, so that the electronic device can perform the video noise reduction function.

[0033] The electronic device can be a personal computer (PC), a cloud device, a mobile device, etc. Mobile devices can be, for example, mobile phones, tablets, personal digital assistants, wearable devices, in-vehicle devices, etc., hardware devices with various operating systems, touchscreens, and / or displays. In the following embodiments, the video noise reduction device is used as an example of an electronic device for illustration.

[0034] like Figure 1 As shown, the video noise reduction method includes the following steps:

[0035] Step 101: Obtain the video to be processed. Each frame of the video includes low-frequency components and Laplacian components.

[0036] In this embodiment of the application, the process of the electronic device performing step 101 can be, for example, acquiring a video to be processed, the video including multiple frames of images; performing k-fold downsampling processing on each frame of the video to obtain low-frequency components in the image; performing k-fold upsampling processing on the low-frequency components to obtain a first upsampling result; and determining the Laplacian component in the image based on the image and the first upsampling result.

[0037] Here, k refers to the single sampling rate. The value of k can be, for example, 2 or 4. The upsampling processing method can be, for example, bilinear interpolation.

[0038] It should be noted that electronic devices can connect to sensors used for video acquisition through interfaces provided by the operating system and drivers, and obtain the video to be processed from the sensors.

[0039] The process by which the electronic device determines the Laplacian component in the image can be, for example, acquiring the difference component between the image and the first upsampling result; and using the difference component as the Laplacian component in the image.

[0040] The formula for obtaining the low-frequency component can be, for example, as shown in formula (1), and the formula for obtaining the Laplace component can be, for example, as shown in formula (2).

[0041] Low-frequency component = downsampled(input image, 1 / k) (1)

[0042] Laplacian component = input image – upsampled(downsampled(input image, 1 / k), k) (2)

[0043] The input image represents each frame of the video.

[0044] Step 102: For each frame of the video, determine the low-frequency component noise reduction result of the image based on the low-frequency component of the image and the low-frequency component noise reduction result of the corresponding previous frame.

[0045] In one embodiment of this application, the input to the denoising neural network can be the low-frequency components of the image and the low-frequency component denoising result of the previous frame image corresponding to the image, and the output can be the low-frequency component denoising result of the image. Correspondingly, the electronic device performing step 102 can, for example, input the low-frequency components of the image and the low-frequency component denoising result of the previous frame image corresponding to the image into the denoising neural network for each frame image in the video, and obtain the low-frequency component denoising result of the image output by the denoising neural network.

[0046] In another example, the input to the denoising neural network can be the low-frequency components of the image and the denoising result of the low-frequency components of the previous frame image corresponding to the image. The output can be denoising-related parameters. Combining the denoising-related parameters, the denoising result of the low-frequency components of the image can be determined. Correspondingly, the process of the electronic device performing step 102 can be, for example, as follows: for each frame image in the video, the low-frequency components of the image and the denoising result of the low-frequency components of the previous frame image corresponding to the image are input into the denoising neural network to obtain the temporal mask and spatial denoising residual output by the denoising neural network; based on the temporal mask, spatial denoising residual, low-frequency components, and the denoising result of the low-frequency components of the previous frame image, the denoising result of the low-frequency components of the image is determined.

[0047] In this embodiment, the denoising neural network can be, for example, a convolutional neural network (CNN), a residual encoder-decoder network (REDNet), or a kernel prediction network (KPN). It should be noted that, in one example, the electronic device can deploy a denoising neural network to obtain the low-frequency component denoising result of the image. In another example, the electronic device can be connected to a neural network accelerator with a deployed denoising neural network, and obtain the low-frequency component denoising result of the image by interacting with the neural network accelerator.

[0048] The formula for determining the low-frequency component noise reduction result of the image can be, for example, as shown in formula (3) below.

[0049] The low-frequency component noise reduction result of the image = temporal mask * (spatial noise reduction residual + low-frequency component of the image) + (1 – temporal mask) * low-frequency component noise reduction result of the previous frame of the image (3)

[0050] In this embodiment of the application, the training method of the denoising neural network may include the following process: acquiring at least one original video and an initial denoising neural network, wherein the original video is acquired in a noise-free environment; adding noise to each frame of the original image in the original video according to a sampling rate of k to obtain a noisy image corresponding to the original image; training the initial denoising neural network based on at least one original video and the noisy image corresponding to the original image in the original video to obtain a trained denoising neural network.

[0051] This includes a noise-free environment, such as a well-lit environment. Since the original image or its corresponding noisy image was not downsampled during training, the process of adding noise to the original image by the electronic device at a sampling rate of k can be, for example, by determining a k*k sampling rate based on the k sampling rate, and then adding noise to the original image according to the k*k sampling rate. The noise involved in the noise addition process, such as Gaussian noise or Poisson noise, can be set according to actual needs.

[0052] Step 103: Determine the intermediate image corresponding to the image based on the low-frequency component noise reduction result and the Laplacian component of the image.

[0053] In this embodiment of the application, the electronic device may perform step 103 as follows: perform k-fold upsampling on the low-frequency component noise reduction result to obtain a second upsampling result; perform fusion processing on the second upsampling result and the Laplacian component to obtain an intermediate image corresponding to the image.

[0054] The electronic device may include an image signal processing (ISP) chip, which may have an ISP interface. The electronic device can use the ISP interface to call the ISP chip to generate intermediate images and perform edge-preserving filtering on the intermediate images.

[0055] The formula for obtaining the intermediate image can be, for example, as shown in formula (4) below.

[0056] Intermediate image = Upsampled (low-frequency component noise reduction result of the image, k) + Laplacian component of the image (4)

[0057] Step 104: Perform edge-preserving filtering on the intermediate image to obtain the corresponding denoised image.

[0058] In this embodiment of the application, the edge-preserving filtering processing method may include at least one of the following: bilateral filtering processing, non-local mean processing, and guided filtering processing.

[0059] The video denoising method of this application embodiment acquires a video to be processed, in which each frame of the video includes a low-frequency component and a Laplacian component. For each frame of the video, the low-frequency component denoising result of the image is determined based on the low-frequency component of the image and the low-frequency component denoising result of the previous frame of the image. Based on the low-frequency component denoising result and the Laplacian component, the corresponding intermediate image is determined. The intermediate image is then subjected to edge-preserving filtering to obtain the denoised image. Thus, by using a denoising neural network to denoise the low-frequency component in the image and then using edge-preserving filtering to denoise the image, the denoising effect of the low-frequency and high-frequency components in the image is ensured, while reducing the computational load.

[0060] Figure 2 This is a schematic flowchart of the video noise reduction method provided in Embodiment 2 of this application. Figure 2 As shown, in Figure 1 Based on the illustrated embodiment, after step 103, the method may further include the following steps:

[0061] Step 105: Use a Bayer luminance channel filter to perform pre-denoising processing on the intermediate image to obtain the pre-denoising result of the intermediate image.

[0062] In this embodiment of the application, the filtering operator of the Bayer luminance channel filter can be, for example, a 3*3 filtering operator, a 5*5 filtering operator, etc.

[0063] Step 106: Determine the motion region confidence mask of the intermediate image based on the pre-denoising result of the intermediate image and the pre-denoising result of the intermediate image corresponding to the previous frame image.

[0064] In this embodiment of the application, the calculation formula for determining the motion region confidence mask of the intermediate image by the electronic device can be, for example, as shown in the following formula (5).

[0065] Motion region confidence mask = absolute value (pre-denoising result of the intermediate image of the image – pre-denoising result of the intermediate image of the previous frame of the image) / threshold (5)

[0066] The threshold is determined based on at least one of the following: ambient brightness during image acquisition, specified noise parameters, and parameters of the acquisition device during image acquisition. It should be noted that a larger specified noise parameter results in a larger threshold. The threshold represents the impact of noise variance on the confidence mask of the moving region.

[0067] The motion region confidence mask can include the motion region confidence score for each pixel in the intermediate image. The motion region confidence mask represents the motion of the pre-denoising result of the intermediate image relative to the pre-denoising result of the intermediate image of the previous frame. For example, for the pre-denoising results corresponding to two adjacent frames, such as the first pre-denoising result and the second pre-denoising result, if the value of a pixel in the first pre-denoising result is the same as the value of a pixel at the corresponding position in the second pre-denoising result, then the motion region confidence score for that pixel is 0.

[0068] Specifically, for each pixel in the first pre-denoising result, the greater the difference between the pixel's value in the first pre-denoising result and the value of the corresponding pixel in the second pre-denoising result, the greater the confidence level of the pixel's motion region. The confidence level of a single pixel's motion region can be any value between 0 and 1.

[0069] Step 107: Determine the temporal denoising result of the intermediate image based on the motion region confidence mask of the intermediate image, the pre-denoising result of the intermediate image, and the temporal denoising result of the intermediate image corresponding to the previous frame image.

[0070] In this embodiment of the application, the electronic device can calculate the temporal noise reduction result of the intermediate image using the following formula (6).

[0071] Temporal denoising result of intermediate image = motion region confidence mask of intermediate image * pre-denoising result of intermediate image + (1 - motion region confidence mask) * temporal denoising result of intermediate image of previous frame (6)

[0072] Step 108: Update the intermediate image based on the temporal denoising results of the intermediate image.

[0073] In this embodiment of the application, the electronic device can directly use the temporal denoising result of the intermediate image as the updated intermediate image.

[0074] The video denoising method of this application embodiment pre-denoises the intermediate image using a Bayer luminance channel filter to obtain a pre-denoising result for the intermediate image; based on the pre-denoising result of the intermediate image and the pre-denoising result of the intermediate image corresponding to the previous frame, a motion region confidence mask for the intermediate image is determined; based on the motion region confidence mask, the pre-denoising result of the intermediate image, and the temporal denoising result of the intermediate image corresponding to the previous frame, a temporal denoising result for the intermediate image is determined; based on the temporal denoising result of the intermediate image, the intermediate image is updated, and then edge-preserving filtering is used to denoise the updated intermediate image, which can further ensure the denoising effect of high-frequency components in the image, thereby improving the denoising effect of the image.

[0075] The following example illustrates this. For example... Figure 3 The image shown is a schematic diagram of video noise reduction. Figure 3 In the process, (1) the current frame image (input image) of the video to be processed is obtained. (2) the current frame image is downsampled by k times to obtain the low frequency component of the current frame image (input image_downsampled). (3) the low frequency component of the current frame image is upsampled by k times to obtain the first upsampled result (input image_downsampled_upsampled). (4) the current frame image and the first upsampled result are subtracted to obtain the Laplacian component. (5) the low frequency component (input image_downsampled) is input into the denoising neural network to obtain the low frequency component denoising result of the image (denoising result). (6) the low frequency component denoising result is upsampled by k times to obtain the second upsampled result (denoising result_upsampled). (7) the Laplacian component and the second upsampled result are fused (added) to obtain the corresponding intermediate image (reconstructed image). (8) the intermediate image is subjected to edge-preserving filtering (traditional denoising module) to obtain the denoised image (output image).

[0076] Figure 4 This is a schematic diagram of the video noise reduction device provided in Embodiment 3 of this application.

[0077] like Figure 4 As shown, the video noise reduction device 1000 includes: an acquisition module 1010, a first determination module 1020, a second determination module 1030, and a filtering processing module 1040.

[0078] The acquisition module 1010 is used to acquire the video to be processed, wherein each frame of the video includes a low-frequency component and a Laplace component.

[0079] The first determining module 1020 is used to determine the low-frequency component noise reduction result of the image for each frame of the video based on the low-frequency component of the image and the low-frequency component noise reduction result of the previous frame corresponding to the image.

[0080] The second determining module 1030 is used to determine the intermediate image corresponding to the image based on the low-frequency component noise reduction result and the Laplacian component of the image.

[0081] The filtering module 1040 is used to perform edge-preserving filtering on the intermediate image to obtain the noise-reduced image corresponding to the image.

[0082] As one possible implementation of this application, the acquisition module 1010 is specifically used to: acquire a video to be processed, the video including multiple frames of images; perform k-fold downsampling on each frame of the video to obtain low-frequency components in the images; perform k-fold upsampling on the low-frequency components to obtain a first upsampling result; and determine the Laplacian component in the images based on the images and the first upsampling result.

[0083] As one possible implementation of this application, the first determining module 1020 is specifically used to: for each frame of the video, input the low-frequency component of the image and the low-frequency component noise reduction result of the previous frame of the image into a noise reduction neural network to obtain the temporal mask and spatial noise reduction residual output by the noise reduction neural network; and determine the low-frequency component noise reduction result of the image based on the temporal mask, the spatial noise reduction residual, the low-frequency component, and the low-frequency component noise reduction result of the previous frame of the image.

[0084] As one possible implementation of this application, the first determining module 1020 is further configured to: acquire at least one original video and an initial denoising neural network, wherein the original video is acquired in a noise-free environment; add noise to each frame of the original image in the original video according to a sampling rate of k to obtain a noisy image corresponding to the original image; and train the initial denoising neural network based on at least one original video and the noisy image corresponding to the original image in the original video to obtain a trained denoising neural network.

[0085] As one possible implementation of this application, the second determining module 1030 is specifically used to perform k-fold upsampling processing on the low-frequency component noise reduction result to obtain a second upsampling result; and to perform fusion processing on the second upsampling result and the Laplacian component to obtain an intermediate image corresponding to the image.

[0086] As a possible implementation of this application embodiment, the apparatus further includes: a pre-denoising module, a third determining module, a fourth determining module, and an updating module; the pre-denoising module is used to perform pre-denoising processing on the intermediate image using a Bayer luminance channel filter to obtain a pre-denoising result of the intermediate image; the third determining module is used to determine a motion region confidence mask of the intermediate image based on the pre-denoising result of the intermediate image and the pre-denoising result of the intermediate image corresponding to the previous frame image; the fourth determining module is used to determine a temporal denoising result of the intermediate image based on the motion region confidence mask of the intermediate image, the pre-denoising result of the intermediate image, and the temporal denoising result of the intermediate image corresponding to the previous frame image; the updating module is used to update the intermediate image based on the temporal denoising result of the intermediate image.

[0087] The video denoising apparatus of this application acquires a video to be processed, wherein each frame of the video includes a low-frequency component and a Laplacian component. For each frame of the video, a low-frequency component denoising result is determined based on the low-frequency component of the image and the low-frequency component denoising result of the previous frame. An intermediate image is determined based on the low-frequency component denoising result and the Laplacian component. The intermediate image is then subjected to edge-preserving filtering to obtain the denoised image. Thus, by using a denoising neural network to denoise the low-frequency component in the image and then using edge-preserving filtering to denoise the image, the denoising effect of both low-frequency and high-frequency components in the image is ensured, while reducing the computational load.

[0088] To achieve the above embodiments, this application also proposes an electronic device, such as... Figure 5 As shown, Figure 5 This is a block diagram illustrating an electronic device for a video noise reduction method according to an exemplary embodiment.

[0089] like Figure 5 As shown, the above-mentioned electronic device 1100 includes:

[0090] The memory 1110 and the processor 1120 are connected by a bus 1130, which connects different components (including the memory 1110 and the processor 1120). The memory 1110 stores a computer program, which implements the video noise reduction method described in this embodiment when the processor 1120 executes the program.

[0091] Bus 1130 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0092] Electronic device 1100 typically includes a variety of electronic device readable media. These media can be any available media that can be accessed by electronic device 1100, including volatile and non-volatile media, removable and non-removable media.

[0093] Memory 1110 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 1140 and / or cache memory 1150. Electronic device 1100 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 1160 may be used to read and write non-removable, non-volatile magnetic media (… Figure 5 Not shown; usually referred to as a "hard drive"). Although Figure 5 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 1130 via one or more data media interfaces. Memory 1110 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this disclosure.

[0094] A program / utility 1180 having a set (at least one) of program modules 1170 may be stored, for example, in memory 1110. Such program modules 1170 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 1170 typically perform the functions and / or methods described in the embodiments of this disclosure.

[0095] Electronic device 1100 can also communicate with one or more external devices 1190 (e.g., keyboard, pointing device, display 1091, etc.), and with one or more devices that enable a user to interact with the electronic device 1100, and / or with any device that enables the electronic device 1100 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 1192. Furthermore, electronic device 1100 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1193. Figure 5 As shown, network adapter 1193 communicates with other modules of electronic device 1100 via bus 1130. It should be understood that, although... Figure 5 As not shown, other hardware and / or software modules can be used in conjunction with electronic device 1100, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0096] The processor 1120 performs various functional applications and data processing by running programs stored in the memory 1110.

[0097] It should be noted that the implementation process and technical principles of the electronic device in this embodiment are explained in the foregoing description of the monitoring and processing method of the present disclosure embodiment, and will not be repeated here.

[0098] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute... Figures 1 to 2 The video noise reduction method described in the embodiments.

[0099] To implement the above embodiments, this disclosure also provides a computer program product, which, when executed by an instruction processor, performs... Figures 1 to 2 The video noise reduction method described in the embodiments.

[0100] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0101] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0102] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0103] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0104] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0105] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0106] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0107] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A video noise reduction method, characterized in that, The method includes: Acquire the video to be processed, which includes multiple frames of images; For each frame of the video, the image is processed to obtain the low-frequency component in the image; The low-frequency component is upsampled by a factor of k to obtain the first upsampling result; Based on the image and the first upsampling result, the Laplacian component in the image is obtained; For each frame of the video, the low-frequency component of the image and the low-frequency component noise reduction result of the previous frame corresponding to the image are input into the noise reduction neural network to obtain the temporal mask and spatial noise reduction residual output by the noise reduction neural network. The low-frequency component noise reduction result of the image is determined based on the temporal mask, the spatial denoising residual, the low-frequency component, and the low-frequency component noise reduction result of the previous frame image. The low-frequency component noise reduction result of the image is upsampled by a factor of k to obtain a second upsampling result; The second upsampling result and the Laplacian component of the image are fused to obtain the intermediate image corresponding to the image. The intermediate image is subjected to edge-preserving filtering to obtain the corresponding denoised image.

2. The method according to claim 1, characterized in that, The training methods for the noise reduction neural network include: At least one original video and an initial denoising neural network are acquired, wherein the original video is acquired in a noise-free environment; For each frame of the original video, noise is added to the original image according to a sampling rate of k to obtain a noisy image corresponding to the original image. The initial denoising neural network is trained based on at least one of the original videos and the noisy images corresponding to the original images in the original videos to obtain a trained denoising neural network.

3. The method according to claim 1, characterized in that, After determining the intermediate image corresponding to the image based on the low-frequency component noise reduction result and the Laplacian component of the image, the method further includes: The intermediate image is pre-denoised using a Bayer luminance channel filter to obtain the pre-denoised result of the intermediate image; Based on the pre-denoising result of the intermediate image and the pre-denoising result of the intermediate image corresponding to the previous frame image, the motion region confidence mask of the intermediate image is determined. The temporal denoising result of the intermediate image is determined based on the motion region confidence mask of the intermediate image, the pre-denoising result of the intermediate image, and the temporal denoising result of the intermediate image corresponding to the previous frame image. The intermediate image is updated based on the temporal denoising results of the intermediate image.

4. A video noise reduction device, characterized in that, The device includes: An acquisition module is used to acquire a video to be processed, the video including multiple frames of images; for each frame of the video, the images are processed to obtain low-frequency components in the images; the low-frequency components are upsampled by a factor of k to obtain a first upsampling result; and the Laplacian components in the images are obtained based on the images and the first upsampling result. The first determining module is used to input the low-frequency components of the image and the low-frequency component noise reduction result of the previous frame image corresponding to the image into a noise reduction neural network for each frame image in the video, to obtain the temporal mask and spatial noise reduction residual output by the noise reduction neural network; and to determine the low-frequency component noise reduction result of the image based on the temporal mask, the spatial noise reduction residual, the low-frequency components and the low-frequency component noise reduction result of the previous frame image. The second determining module is used to perform k-fold upsampling on the low-frequency component noise reduction result of the image to obtain a second upsampling result; and to perform fusion processing on the second upsampling result and the Laplacian component of the image to obtain an intermediate image corresponding to the image. The filtering module is used to perform edge-preserving filtering on the intermediate image to obtain the noise-reduced image corresponding to the image.

5. The apparatus according to claim 4, characterized in that, The first determining module is further specifically used for, At least one original video and an initial denoising neural network are acquired, wherein the original video is acquired in a noise-free environment; For each frame of the original video, noise is added to the original image according to a sampling rate of k to obtain a noisy image corresponding to the original image. The initial denoising neural network is trained based on at least one of the original videos and the noisy images corresponding to the original images in the original videos to obtain a trained denoising neural network.

6. The apparatus according to claim 4, characterized in that, The device further includes: a pre-noise reduction module, a third determination module, a fourth determination module, and an update module; The pre-denoising module is used to perform pre-denoising processing on the intermediate image using a Bayer luminance channel filter to obtain the pre-denoising result of the intermediate image; The third determining module is used to determine the motion region confidence mask of the intermediate image based on the pre-denoising result of the intermediate image and the pre-denoising result of the intermediate image corresponding to the previous frame image. The fourth determining module is used to determine the temporal denoising result of the intermediate image based on the motion region confidence mask of the intermediate image, the pre-denoising result of the intermediate image, and the temporal denoising result of the intermediate image corresponding to the previous frame image. The update module is used to update the intermediate image based on the temporal noise reduction result of the intermediate image.

7. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-3.

8. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-3.

Images