Video noise reduction method, electronic device, storage medium and computer program product

Abstract

The embodiment of the invention discloses a video noise reduction method, an electronic device, a storage medium and a computer program product. The electronic device determines image difference data based on a current frame image of a video and a previous frame image after noise reduction; performing first noise reduction processing on the current frame image based on the image difference data to obtain a first noise reduction image; and performing fusion processing on the first denoised image and the denoised previous frame image to obtain a denoised current frame image.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a video noise reduction method, an electronic device, a storage medium and a computer program product. Background Art

[0002] At present, when denoising videos, some denoising algorithms are mostly used, such as neighborhood filtering or non-local average algorithm, but the videos denoised by these methods still have problems such as loss of texture details or more residual noise; and some video denoising methods based on deep learning require training of multi-frame time domain networks, training data sets are difficult to collect, and the training effect is poor, resulting in poor denoising effect; it can be seen that how to improve the denoising effect of video denoising is a problem that needs to be solved urgently. Summary of the invention

[0003] The embodiments of the present application provide a video noise reduction method, an electronic device, a storage medium, and a computer program product, which can effectively improve the noise reduction effect of video noise reduction.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides a video noise reduction method, the method comprising:

[0006] Determine image difference data based on a current frame image of the video and a previous frame image after noise reduction;

[0007] Performing a first noise reduction process on the current frame image based on the image difference data to obtain a first noise reduction image;

[0008] The first denoised image and the denoised previous frame image are fused to obtain a denoised current frame image.

[0009] In a second aspect, an embodiment of the present application provides an electronic device, the electronic device comprising a determination unit, a noise reduction unit, and a fusion unit;

[0010] A determination unit, configured to determine image difference data based on a current frame image of a video and a previous frame image after noise reduction;

[0011] A denoising unit, configured to perform a first denoising process on the current frame image based on the image difference data to obtain a first denoised image;

[0012] The fusion unit is used to fuse the first denoised image and the previous frame image after denoising to obtain the current frame image after denoising.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory storing instructions executable by the processor; when the instructions are executed by the processor, the above-mentioned video noise reduction method is implemented.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned video noise reduction method when executed by a processor.

[0015] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the steps in the above video noise reduction method.

[0016] The embodiments of the present application provide a video denoising method, an electronic device, a storage medium and a computer program product. The electronic device determines image difference data based on a current frame image of a video and a previous frame image after denoising; performs a first denoising process on the current frame image based on the image difference data to obtain a first denoised image; and performs a fusion process on the first denoised image and the previous frame image after denoising to obtain a current frame image after denoising. It can be seen that in the embodiments of the present application, when the electronic device performs noise reduction on the video, it can perform noise reduction on each current frame image in the video, thereby completing the noise reduction of the entire video, and when performing noise reduction on the current frame image, it can first obtain the previous frame image after noise reduction, determine the image difference data between the current frame image and the previous frame image after noise reduction, and use the image difference data to perform the first noise reduction processing on the current frame image, thereby realizing spatial noise reduction of the current frame image based on the image difference data; then, the obtained first noise reduction image and the previous frame image after noise reduction are fused in the time domain, which can realize a joint noise reduction scheme of spatial and temporal domains, so that the final noise reduction current frame image can better retain the texture while removing the noise, reduce the "ghost" problem that occurs during noise reduction, and finally obtain the noise reduction result of the video from the current frame images after noise reduction, which can greatly improve the noise reduction effect of the video noise reduction. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Schematic diagram of the implementation process of the video noise reduction method proposed in the embodiment of the present application Figure 1 ;

[0018] Figure 2 Schematic diagram of the implementation process of the video noise reduction method proposed in the embodiment of the present application Figure 2 ;

[0019] Figure 3 Schematic diagram of the implementation process of the video noise reduction method proposed in the embodiment of the present application Figure 3 ;

[0020] Figure 4 A schematic diagram of a trained network proposed in an embodiment of the present application;

[0021] Figure 5 Schematic diagram of the structure of the electronic device proposed in the embodiment of the present application Figure 1 ;

[0022] Figure 6 Schematic diagram of the structure of the electronic device proposed in the embodiment of the present application Figure 2 . DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. It is understood that the specific embodiments described herein are only used to explain the related applications, rather than to limit the applications. It should also be noted that, for ease of description, only the parts related to the related applications are shown in the drawings.

[0024] Among the current methods for video denoising, neighborhood filtering or non-local average algorithms are generally used to implement traditional spatial denoising; and traditional temporal denoising schemes are generally implemented by finding similar blocks of adjacent frames based on motion vectors and then performing temporal superposition; however, these traditional video denoising schemes do not perform well. After using traditional video denoising schemes, the video will have problems such as loss of texture details or more residual noise. In addition, there are also video denoising schemes based on deep learning, but multi-frame video denoising schemes based on deep learning require training of multi-frame temporal networks. The temporal data sets required for these trainings are difficult to collect, which will make it difficult for the network to train for better results; temporal neural networks often have problems such as large computational complexity and inconvenient debugging, and because the denoising results of neural networks have a certain degree of randomness, using pure network denoising schemes will result in unpredictable denoising results in some scenarios; it can be seen that the current related video denoising methods generally have the problem of poor denoising effects.

[0025] In order to solve the above problems, in an embodiment of the present application, the electronic device can determine image difference data based on a current frame image of a video and a previous frame image after noise reduction; perform a first noise reduction process on the current frame image based on the image difference data to obtain a first noise reduction image; and perform a fusion process on the first noise reduction image and the previous frame image after noise reduction to obtain a current frame image after noise reduction, thereby effectively improving the noise reduction effect of the video noise reduction.

[0026] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0027] Figure 1 Schematic diagram of the implementation process of the video noise reduction method proposed in the embodiment of the present application Figure 1,like Figure 1 As shown, in an embodiment of the present application, a method for an electronic device to reduce noise on a video may include the following steps:

[0028] Step 101: Determine image difference data based on a current frame image of a video and a previous frame image after noise reduction.

[0029] In an embodiment of the present application, the electronic device may first determine image difference data based on a current frame image of a video and a previous frame image after noise reduction.

[0030] It should be noted that in the embodiments of the present application, the electronic device can be any electronic device with communication and storage functions, such as: mobile phones, personal computers (PCs), tablet computers, e-readers, laptop computers, vehicle-mounted electronic devices, network televisions, wearable electronic devices, personal digital assistants (PDAs), portable media players (PMPs), navigation devices and other electronic devices.

[0031] In some embodiments of the present application, the electronic device may include a visual sensor, such as a camera or a webcam component for shooting videos, and the electronic device may perform video noise reduction on the video shot by its built-in visual sensor.

[0032] In an embodiment of the present application, the electronic device may perform noise reduction on each frame of a video, thereby obtaining a noise-reduced video based on each noise-reduced frame.

[0033] In some embodiments of the present application, the electronic device may decompose the video into individual frames, thereby performing a video noise reduction method on each frame image.

[0034] In an embodiment of the present application, the current frame image represents the frame image currently undergoing noise reduction by the electronic device; the previous frame image after noise reduction represents the frame image after noise reduction corresponding to the previous frame image of the current frame image.

[0035] In an embodiment of the present application, the image difference data represents the difference data between the current frame image and the previous frame image after noise reduction. Since there may be changes or motions between each frame image in the video, the image difference data can also be understood as an image that can reflect the motion differences between frame images.

[0036] In some embodiments of the present application, when the electronic device determines image difference data based on a current frame image of a video and a previous frame image after denoising, it can perform a second denoising process on the current frame image to obtain a second denoised image; perform a pixel block division process on the previous frame image after denoising to obtain a division result; perform a pixel block matching process on the division result and the second denoised image to obtain a pixel block matching result; and determine the image difference data based on the pixel block matching result.

[0037] In an embodiment of the present application, the second noise reduction processing can be a preliminary noise reduction processing. The electronic device can implement the second noise reduction processing through some simple noise reduction algorithms. For example, a bilateral filtering algorithm or a non-local mean (NLM) algorithm can be used to perform a second noise reduction processing on the current frame image to reduce the noise difference between the current frame and the previous frame image after noise reduction, thereby helping to improve the efficiency of determining image difference data.

[0038] In some embodiments of the present application, the pixel block matching result can be used to indicate the second pixel blocks in the second denoised image that match the first pixel blocks of each division in the division result; through the pixel block matching result, the previous frame image after denoising and the second denoised image can be aligned.

[0039] In some embodiments of the present application, when the electronic device performs pixel block division processing on the previous frame image after denoising and obtains the division result, the electronic device may perform pixel block division processing on the previous frame image after denoising according to a preset pixel block size, thereby dividing the previous frame image after denoising into a plurality of first pixel blocks whose sizes meet the preset pixel block size, and obtaining the division result from these first pixel blocks; wherein the specific value of the preset pixel block size is not limited in the present application, for example, the preset pixel block size may be N×N.

[0040] In some embodiments of the present application, when the electronic device performs pixel block matching processing on the division result and the second denoised image to obtain a pixel block matching result, the second denoised image can be used as a reference frame, and a search window can be used to move in the reference frame to determine the second pixel block that best matches the first pixel block within the area of ​​the search window according to a minimum mean square error matching rule, thereby obtaining a pixel block matching result based on the mutually matching first pixel block and second pixel block.

[0041] In some embodiments of the present application, after obtaining the second denoised image, the electronic device may also use a Gaussian pyramid decomposition method to determine a pixel block matching result between the second denoised image and the previous frame image after denoising; the electronic device may convert the second denoised image and the previous frame image after denoising into representations of different resolutions, the bottom of the Gaussian pyramid is a high-resolution representation of the second denoised image and the previous frame image after denoising, and the top is a low-resolution approximation of the second denoised image and the previous frame image after denoising, and then starting from the top of the Gaussian pyramid, the second denoised image with the lowest resolution and the previous frame image after denoising are matched in pixel blocks. Since the resolution is the lowest, the processing speed is relatively fast. The matching process is fast, but the matching accuracy is low. After obtaining the first matching result between the second denoised image with the lowest resolution and the previous frame image after denoising, the electronic device can determine the search area for performing the next pixel block matching according to the first matching result. For example, the search area can be the area in the second denoised image with a higher resolution corresponding to the second layer of the Gaussian pyramid, thereby continuing to determine the second matching result at this resolution, and determining the search area of ​​the next layer according to the second matching result, until the pixel block matching result of the second denoised image represented by the bottom high resolution and the previous frame image after denoising is obtained; based on this layer-by-layer matching from low resolution to high resolution, the processing efficiency of pixel block matching can be improved.

[0042] In some embodiments of the present application, when the electronic device determines the image difference data based on the pixel block matching result, it can determine the initial difference data according to the difference between the first pixel block and the second pixel block; and then perform image smoothing processing on the initial difference data to obtain the image difference data.

[0043] In the embodiments of the present application, the specific method used for image smoothing is not limited in the present application. For example, a Gaussian filter smoothing method can be used to perform image smoothing on the initial difference data to obtain image difference data.

[0044] It can be understood that, in the embodiment of the present application, the initial difference data is obtained by performing a difference operation on each first pixel block and second pixel block that match each other.

[0045] In some embodiments of the present application, when the electronic device determines image difference data based on pixel block matching results, it may also perform morphological processing on the initial difference data after determining the initial difference data according to the difference between the first pixel block and the second pixel block, so as to expand the size of the motion area in the initial difference data so that the processed difference data can contain more motion information, and then perform image smoothing processing on the processed difference data to obtain image difference data.

[0046] Step 102: Perform a first noise reduction process on the current frame image based on the image difference data to obtain a first noise reduction image.

[0047] In an embodiment of the present application, after determining image difference data based on a current frame image of a video and a previous frame image after noise reduction, the electronic device may perform a first noise reduction process on the current frame image based on the image difference data to obtain a first noise reduced image.

[0048] In the embodiment of the present application, the first noise reduction processing can be understood as a spatial noise reduction processing.

[0049] In some embodiments of the present application, when the electronic device performs a first noise reduction process on the current frame image based on the image difference data to obtain a first noise reduction image, the electronic device may include the following steps:

[0050] Step 102a: Determine a noise reduction intensity parameter based on the image difference data.

[0051] In an embodiment of the present application, when the electronic device performs a first noise reduction process on the current frame image based on the image difference data to obtain a first noise reduction image, the electronic device may first determine a noise reduction intensity parameter based on the image difference data.

[0052] In an embodiment of the present application, the noise reduction intensity parameter may be used to control the noise reduction intensity of a subsequent first noise reduction process in the spatial domain.

[0053] In some embodiments of the present application, when the electronic device determines the noise reduction intensity parameter based on the image difference data, the electronic device may determine the noise reduction intensity parameter according to a first value when the pixel value of the image difference data is less than a first threshold; determine the noise reduction intensity parameter according to a second value when the pixel value of the image difference data is greater than or equal to the first threshold and less than or equal to a second threshold; and determine the noise reduction intensity parameter according to a third value when the pixel value of the image difference data is greater than the second threshold.

[0054] In the embodiment of the present application, since the image difference data may be an image that can reflect motion changes, the noise reduction intensity parameter may be determined by the pixel value of the image difference data.

[0055] In the embodiments of the present application, the first threshold, the second threshold, the first value, the second value and the third value are not limited in the present application.

[0056] In some embodiments of the present application, the electronic device may determine the second value according to the first preset parameter and the pixel value of the image difference data.

[0057] Exemplarily, the above method of determining the noise reduction strength parameter based on the image difference data can be expressed as the following formula:

[0058]

[0059] Wherein, w represents the noise reduction strength parameter, the first value is 1, the second value is α(1-m), and the third value is 0; m represents the pixel value of the image difference data; T l1 represents the first threshold, T h1 represents the second threshold; in the second value, α represents the first preset parameter.

[0060] Step 102b: Perform a first denoising process on the current frame image based on the denoising intensity parameter and the trained network to obtain a first denoised image.

[0061] In an embodiment of the present application, when the electronic device performs a first noise reduction processing on the current frame image based on the image difference data to obtain a first noise reduced image, it can determine a noise reduction strength parameter based on the image difference data, and then perform a first noise reduction processing on the current frame image based on the noise reduction strength parameter to obtain the first noise reduced image.

[0062] In an embodiment of the present application, the electronic device performs the first noise reduction processing based on the noise reduction intensity parameter, and can control the noise reduction intensity of the first noise reduction processing, so that the first noise reduction processing can be performed at a noise reduction intensity adapted to the current frame image.

[0063] In some embodiments of the present application, when the electronic device performs a first denoising process on the current frame image based on the denoising intensity parameter and the trained network to obtain the first denoised image, it can perform a feature extraction process on the current frame image based on the feature extraction layer in the trained network to obtain the first image data; and then perform a first denoising process on the first image data based on the denoising intensity parameter and the noise residual in the trained network to obtain the first denoised image.

[0064] In an embodiment of the present application, the trained network may be a trained neural network. The specific structure of the neural network is not limited in this application. For example, the feature extraction layer may include modules such as convolution, downsampling, residual connection, and upsampling. Static-to-Dynamic (S2D), dynamic-to-static (D2S) and other operators may be added on the basis of these basic modules. In addition, multiplication operators and addition operators may be set before the output layer to control the noise reduction results.

[0065] In an embodiment of the present application, a neural network can be trained, and the purpose of the training is to determine the noise residual of the neural network; the trained network can post the noise residual back to the first image data, thereby realizing the noise reduction function of the current frame image. In this process, the noise reduction intensity parameter can be used to control the realization of different proportions of noise residual.

[0066] In an embodiment of the present application, the first image data may include feature data of noise extracted by a trained network from the current frame image.

[0067] Exemplarily, the method for determining the first denoised image using the trained network can be expressed as the following formula:

[0068] I out =w×I res +I in (2)

[0069] Among them, I out It is the first denoised image output by the trained network, w is the denoising intensity parameter, I res represents the noise residual, I in represents the current frame image of the network after input training; the above method can ensure the effective use of the original image information, such as when w = 0, I out =I in , that is, the original image information can be transmitted backwards without causing the information to be lost midway.

[0070] In some embodiments of the present application, the electronic device may also obtain a second image parameter of the current frame image, determine a local noise reduction parameter according to the second image parameter, and perform local noise reduction processing on the current frame image using the local noise reduction parameter.

[0071] In an embodiment of the present application, the second image parameter may include at least one of brightness information, color information, and a face detection result.

[0072] In an embodiment of the present application, brightness information can be used to indicate the brightness of different areas in the current frame image; color information can be used to indicate the color in the current frame image, as well as the distribution of the color; and face detection results can be used to indicate whether there is a face area in the current frame image.

[0073] In some embodiments of the present application, the local noise reduction parameter may be used to indicate the noise reduction intensity of the local area in the current frame image.

[0074] In some embodiments of the present application, the electronic device may use a first local noise reduction parameter to perform local noise reduction processing on an area in the current frame image whose brightness is greater than a preset brightness threshold; wherein the specific value of the first local noise reduction parameter is not limited in this application.

[0075] It should be noted that, in an embodiment of the present application, for areas where the brightness is greater than or equal to a preset brightness threshold, the noise reduction intensity can be appropriately reduced. Therefore, the first local noise reduction parameter can be smaller than the second local noise reduction parameter; wherein the second local noise reduction parameter represents the local noise reduction parameter corresponding to the brightness less than the preset brightness threshold.

[0076] In some embodiments of the present application, the electronic device may use a third local noise reduction parameter to perform local noise reduction processing on an area in the current frame image whose color information meets a preset color condition; wherein the specific value of the third local noise reduction parameter is not limited in this application.

[0077] It should be noted that, in an embodiment of the present application, an area that meets the preset color conditions indicates that the area is rich in color; for an area that meets the preset color conditions, the noise reduction intensity can be appropriately reduced, and therefore, the third local noise reduction parameter can be smaller than the fourth local noise reduction parameter; wherein the fourth local noise reduction parameter represents the local noise reduction parameter corresponding to the color information that does not meet the preset color conditions.

[0078] In some embodiments of the present application, the electronic device may use a fifth local noise reduction parameter to perform local noise reduction processing on the area where the face detection result of the current frame image indicates that a face exists; wherein the specific value of the fifth local noise reduction parameter is not limited in this application.

[0079] It should be noted that in an embodiment of the present application, for areas where faces exist, the noise reduction intensity can be appropriately reduced, and therefore, the fifth local noise reduction parameter can be smaller than the sixth local noise reduction parameter; wherein the sixth local noise reduction parameter represents the local noise reduction parameter corresponding to the area where no faces exist in the current frame image.

[0080] In some embodiments of the present application, the present application does not limit the execution order of the electronic device using local noise reduction parameters to perform local noise reduction processing on the current frame image. For example, the electronic device may perform a first noise reduction processing on the current frame image based on the noise reduction strength parameters to obtain a first noise reduction image, and then perform local noise reduction processing on the current frame image using the local noise reduction parameters to obtain a third noise reduction image; it may also perform a first noise reduction processing on the current frame image based on the noise reduction strength parameters to obtain the first noise reduction image, and at the same time perform local noise reduction processing on the current frame image using the local noise reduction parameters to obtain a third noise reduction image; it may also perform a first noise reduction processing on the current frame image based on the noise reduction strength parameters to obtain the first noise reduction image before obtaining the first noise reduction image, and then perform local noise reduction processing on the current frame image using the local noise reduction parameters to obtain a third noise reduction image.

[0081] Step 103: perform fusion processing on the first denoised image and the previous frame image after denoising to obtain a current frame image after denoising.

[0082] In an embodiment of the present application, the electronic device can perform a first noise reduction process on the current frame image based on the image difference data to obtain a first noise reduction image, and then perform a fusion process on the first noise reduction image and the previous frame image after noise reduction to obtain a current frame image after noise reduction.

[0083] It can be understood that, in the embodiment of the present application, the electronic device can determine the noise reduction result of the video based on the noise-reduced current frame images corresponding to each current frame image in the video.

[0084] In an embodiment of the present application, by fusing the first denoised image and the previous frame image after denoising in the time domain, a denoising scheme combining spatial and temporal domains can be implemented on the basis of the first denoised image previously obtained in the spatial domain. Compared with the current related video denoising methods, the scheme can better retain the texture while removing noise, and can reduce the problems such as "ghosting" of traditional time domain denoising, with better overall effect and higher adjustability. Among them, the "ghosting" problem refers to the existence of one or more images similar to the main object but slightly offset in position in addition to the main object in the image.

[0085] In some embodiments of the present application, when the electronic device fuses the first denoised image and the previous denoised frame image to obtain the denoised current frame image, the electronic device can determine fusion parameters based on image difference data; and then fuse the first denoised image and the previous denoised frame image based on the fusion parameters to obtain the denoised current frame image.

[0086] In some embodiments of the present application, when the electronic device determines the fusion parameters based on the image difference data, the electronic device may determine the fusion parameters according to a fourth value when the pixel value of the image difference data is less than a third threshold; determine the fusion parameters according to a fifth value when the pixel value of the image difference data is greater than or equal to the third threshold and less than or equal to the fourth threshold; and determine the fusion parameters according to a sixth value when the pixel value of the image difference data is greater than the fourth threshold.

[0087] In the embodiments of the present application, the specific values ​​of the third threshold, the fourth threshold, the fourth value, the fifth value and the sixth value are not limited in the present application.

[0088] In some embodiments of the present application, the electronic device may determine the fifth value according to the second preset parameter and the pixel value of the image difference data.

[0089] Exemplarily, the method for determining fusion parameters based on image difference data can be expressed as the following formula:

[0090]

[0091] Wherein, γ represents the fusion parameter, the fourth value is 1, the fifth value is β(1-m), and the sixth value is 0; m represents the pixel value of the image difference data, T l2 represents the third threshold, T h2 represents the fourth threshold; β represents the second preset parameter.

[0092] In some embodiments of the present application, the electronic device may further include a gyroscope. When determining the fusion parameters, the electronic device may not only determine the fusion parameters based on the image difference data, but may also obtain the data of the gyroscope and judge the motion state based on the data of the gyroscope. When the electronic device determines that the motion state meets the preset motion conditions, the fusion parameters may be set to 0.

[0093] In some embodiments of the present application, the electronic device may determine a first threshold, a second threshold, a third threshold, a fourth threshold, a first preset parameter, and a second preset parameter based on a first image parameter; wherein the first image parameter includes at least one of sensitivity (International Organization for Standardization, ISO) information, dynamic range information, scene detection results, and color temperature information of the current frame image.

[0094] In an embodiment of the present application, the photosensitivity information may be used to measure the sensitivity of a photosensitive element of a camera for acquiring a video to light.

[0095] In an embodiment of the present application, the dynamic range information can be used to represent the brightness difference range between the darkest and brightest parts displayed or captured in the current frame image, and can be used to describe the measurement of how many different brightness levels are presented in the current frame image.

[0096] In an embodiment of the present application, the scene detection result may be a result of scene detection on the current frame image. For example, the scene detection result may be a detection result of natural scenery, indoors, portraits, and animals.

[0097] In an embodiment of the present application, the color temperature information may be information on the characteristic color of light emitted by a light source in a current frame image.

[0098] It should be noted that in an embodiment of the present application, in order to improve the noise reduction effect of the video, the first image parameter of the frame image in the video can be obtained, and the corresponding first threshold, second threshold, third threshold, fourth threshold, first preset parameter and second preset parameter can be set according to the first image parameter, so as to improve the processing effects of spatial noise reduction and time domain fusion.

[0099] In some embodiments of the present application, different parameter measurement conditions can be set in advance, and the parameter measurement conditions may include measurement conditions of sensitivity information, dynamic range information, scene detection results, and color temperature information, and then respectively set corresponding first thresholds, second thresholds, third thresholds, fourth thresholds, first preset parameters, and second preset parameters for situations that meet the measurement conditions and situations that do not meet the measurement conditions.

[0100] Exemplarily, for sensitivity information, a measurement condition for measuring the sensitivity information is set, for example, the measurement condition is a sensitivity threshold. When the sensitivity information is greater than the sensitivity threshold, it means that the sensitivity of the current frame image is relatively high. In order to obtain the noise reduction effect adapted to the frame image with higher sensitivity, it is necessary to set a lower noise reduction intensity parameter. Therefore, for the sensitivity that meets this situation, the first threshold, the second threshold, the third threshold, the fourth threshold, the first preset parameter and the second preset parameter can be set accordingly, so that when the pixel value based on the image difference data is compared with the first threshold and the second threshold, and the noise reduction intensity parameter is determined according to the first comparison result, a smaller noise reduction intensity parameter can be obtained relative to the noise reduction intensity parameter corresponding to the lower sensitivity information.

[0101] In some embodiments of the present application, when a first denoised image and a previous denoised frame image are fused based on fusion parameters to obtain a current denoised frame image, first data to be processed can be determined based on the fusion parameters and the first denoised image, second data to be processed can be determined based on the fusion parameters and the previous denoised frame image, and then the current denoised frame image can be determined based on the first data to be processed and the second data to be processed.

[0102] Exemplarily, a method of fusing the first denoised image and the previous denoised frame image based on the fusion parameter to obtain the denoised current frame image can be expressed as the following formula:

[0103] I final =γ×I out +(1-γ)×I pre (4)

[0104] Among them, I final represents the current frame image after denoising, γ represents the fusion parameter, I out represents the first denoised image output by the trained network, I pre Represents the previous frame image after denoising.

[0105] An embodiment of the present application provides a video denoising method, in which an electronic device determines image difference data based on a current frame image of a video and a previous frame image after denoising; performs a first denoising process on the current frame image based on the image difference data to obtain a first denoised image; and performs a fusion process on the first denoised image and the previous frame image after denoising to obtain a current frame image after denoising. It can be seen that in the embodiments of the present application, when the electronic device performs noise reduction on the video, it can perform noise reduction on each current frame image in the video, thereby completing the noise reduction of the entire video, and when performing noise reduction on the current frame image, it can first obtain the previous frame image after noise reduction, determine the image difference data between the current frame image and the previous frame image after noise reduction, and use the image difference data to perform the first noise reduction processing on the current frame image, thereby realizing spatial noise reduction of the current frame image based on the image difference data; then, the obtained first noise reduction image and the previous frame image after noise reduction are fused in the time domain, which can realize a joint noise reduction scheme of spatial and temporal domains, so that the final noise reduction current frame image can better retain the texture while removing the noise, reduce the "ghost" problem that occurs during noise reduction, and finally obtain the noise reduction result of the video from the current frame images after noise reduction, which can greatly improve the noise reduction effect of the video noise reduction.

[0106] Based on the above embodiment, in another embodiment of the present application, Figure 2 As shown, when the electronic device performs video noise reduction on a video, the electronic device may further include the following steps:

[0107] Step 104: determine a local noise reduction parameter according to a second image parameter; wherein the second image parameter includes at least one of brightness information, color information, and a face detection result of the current frame image.

[0108] In some embodiments of the present application, the present application does not limit the execution order of the electronic device determining the local noise reduction parameters according to the second image parameters. For example, the electronic device may perform a first noise reduction process on the current frame image based on the image difference data to obtain a first noise reduction image, that is, after step 102, and then determine the local noise reduction parameters according to the second image parameters, such as Figure 2 The execution order shown; while performing the first denoising process on the current frame image based on the denoising intensity parameter to obtain the first denoised image, the local denoising parameters can be determined according to the second image parameters; and before performing the first denoising process on the current frame image based on the denoising intensity parameter to obtain the first denoised image, the local denoising parameters can be determined according to the second image parameters.

[0109] In an embodiment of the present application, brightness information can be used to indicate the brightness of different areas in the current frame image; color information can be used to indicate the color in the current frame image, as well as the distribution of the color; and face detection results can be used to indicate whether there is a face area in the current frame image.

[0110] In an embodiment of the present application, the local noise reduction parameter can be used to indicate the noise reduction intensity of the local area in the current frame image, so that the local area can be denoised according to the noise reduction intensity suitable for the local area, thereby improving the picture quality of the local area and improving the overall noise reduction effect of the video.

[0111] In an embodiment of the present application, the local noise reduction parameters may include a first local noise reduction parameter, a second local noise reduction parameter, a third local noise reduction parameter, a fourth local noise reduction parameter, a fifth local noise reduction parameter and a sixth local noise reduction parameter; wherein the first local noise reduction parameter may be smaller than the second local noise reduction parameter, the third local noise reduction parameter may be smaller than the fourth local noise reduction parameter, and the fifth local noise reduction parameter may be smaller than the sixth local noise reduction parameter.

[0112] In some embodiments of the present application, for an area in the current frame image whose brightness information is greater than or equal to a preset brightness threshold, the electronic device can determine the local noise reduction parameter to be a first local noise reduction parameter; for an area in the current frame image whose brightness information is less than a preset brightness threshold, the electronic device can determine the local noise reduction processing to be a second local noise reduction parameter.

[0113] It should be noted that, in the embodiment of the present application, for an area whose brightness is greater than or equal to a preset brightness threshold, the noise reduction intensity can be appropriately reduced, and therefore, the first local noise reduction parameter can be smaller than the second local noise reduction parameter.

[0114] In some embodiments of the present application, for an area in the current frame image where color information meets a preset color condition, the electronic device may determine that the local noise reduction parameter is a third local noise reduction parameter; for an area in the current frame image where color information does not meet a preset color condition, the electronic device may determine that the local noise reduction processing is a fourth local noise reduction parameter.

[0115] It should be noted that, in the embodiment of the present application, an area that meets the preset color conditions indicates that the area is rich in color; for an area that meets the preset color conditions, the noise reduction intensity can be appropriately reduced, and therefore, the third local noise reduction parameter can be smaller than the fourth local noise reduction parameter.

[0116] In some embodiments of the present application, for an area where a face is indicated by a face detection result of the current frame image, the electronic device may determine the local noise reduction parameter to be the fifth local noise reduction parameter; for an area where a face detection result of the current frame image indicates that a face is not present, the electronic device may determine the local noise reduction parameter to be the sixth local noise reduction parameter.

[0117] It should be noted that, in the embodiment of the present application, for the area where the face exists, the noise reduction intensity can be appropriately reduced, and therefore, the fifth local noise reduction parameter can be smaller than the sixth local noise reduction parameter.

[0118] Step 105: Perform local noise reduction processing on the current frame image according to the local noise reduction parameters to obtain a third noise reduction image.

[0119] In an embodiment of the present application, after determining the local noise reduction parameters according to the second image parameters, the electronic device may perform local noise reduction processing on the current frame image according to the local noise reduction parameters to obtain a third noise reduction image.

[0120] In some embodiments of the present application, the electronic device performs local noise reduction processing on the current frame image according to local noise reduction parameters, and the method for obtaining a third noise reduction image may include at least one of the following: using the first local noise reduction parameter to perform local noise reduction processing on an area in the current frame image whose brightness is greater than a preset brightness threshold, to obtain a third noise reduction image; using the third local noise reduction parameter to perform local noise reduction processing on an area in the current frame image whose color information meets a preset color condition, to obtain a third noise reduction image; using the fifth local noise reduction parameter to perform local noise reduction processing on an area in the current frame image where a face detection result indicates that a face exists, to obtain a third noise reduction image.

[0121] Step 106: perform fusion processing on the first denoised image, the third denoised image and the previous frame image after denoising to obtain a current frame image after denoising.

[0122] In an embodiment of the present application, after the electronic device performs local noise reduction processing on the current frame image according to the local noise reduction parameters to obtain the third noise reduction image, it can fuse the first noise reduction image, the third noise reduction image and the previous frame image after noise reduction to obtain the current frame image after noise reduction.

[0123] In an embodiment of the present application, the electronic device performs a first noise reduction process on the current frame image based on a noise reduction intensity parameter to obtain a first noise reduction image, and performs a local noise reduction process on the current frame image according to a local noise reduction parameter to obtain a third noise reduction image. Then, the electronic device can perform a fusion process on the first noise reduction image, the third noise reduction image, and the previous frame image after noise reduction to obtain the current frame image after noise reduction.

[0124] In some embodiments of the present application, when the first denoised image, the third denoised image and the previous frame image after denoising are fused to obtain the current frame image after denoising, the weights corresponding to the first denoised image, the third denoised image and the previous frame image after denoising can be set respectively, so that the first denoised image, the third denoised image and the previous frame image after denoising are fused according to the weights corresponding to the first denoised image, the third denoised image and the previous frame image after denoising to obtain the current frame image after denoising.

[0125] The embodiment of the present application provides a video denoising method, wherein the electronic device may perform a first denoising process on a current frame image based on image difference data to obtain a first denoised image, and then determine a local denoising parameter according to a second image parameter; wherein the second image parameter includes at least one of brightness information, color information and face detection result of the current frame image; the current frame image is locally denoised according to the local denoising parameter to obtain a third denoised image, and the first denoised image, the third denoised image and the previous denoised frame image are fused to obtain the denoised current frame image; thus, in the embodiment of the present application, the electronic device may perform adaptive local denoising on the current frame image in addition to obtaining the first denoised image when performing video denoising, and finally fuse the result of the local denoising with the first denoised image and the previous denoised frame image, so as to perform adaptive denoising on the current frame image in different situations, thereby constituting a denoising result of a complete video according to each denoised current frame image, and thus greatly improving the denoising effect of the video.

[0126] Based on the above embodiment, in another embodiment of the present application, exemplarily, the electronic device is a mobile phone, and video noise reduction can be performed on a video captured by its built-in camera; Figure 3 As shown, when the mobile phone performs video noise reduction on a video, it can perform a second noise reduction process on the current frame image of the video to obtain a second noise reduction image (step 201), and then use the previous frame image after noise reduction and the second noise reduction image to determine image difference data (step 202), and determine the noise reduction intensity parameter according to the image difference data (step 203), and then use the noise reduction intensity parameter to perform a first noise reduction process on the current frame image to obtain a first noise reduction image (step 204), and then use the image difference data to determine the fusion parameter (step 205), and finally use the fusion parameter to perform a fusion process on the first noise reduction image and the previous frame image after noise reduction to obtain the current frame image after noise reduction (step 206).

[0127] In some embodiments of the present application, when a second noise reduction process is performed on a current frame image of a video to obtain a second noise reduction image, the second noise reduction process can be understood as preliminary noise reduction, which can reduce the noise difference between the current frame and the previous frame image after noise reduction, so as to better align the images; the embodiments of the present application do not limit the noise reduction algorithm of the second noise reduction process. When the computing resources of the electronic device are insufficient, simple bilateral filtering can be used. When the computing resources are sufficient, other complex noise reduction algorithms, such as the NLM algorithm, can also be used.

[0128] In some embodiments of the present application, a block matching algorithm can be used to align the previous frame image after denoising to the current frame image; for example, the current frame that has undergone preliminary denoising, that is, the second denoised image, can be used as a reference frame, and then the previous frame image after denoising can be divided into several pixel blocks, and then each pixel block is searched for the pixel block that best matches it in a specific search area of ​​the reference frame in turn, and the matching and alignment are carried out in units of pixel blocks.

[0129] Exemplarily, the electronic device can select a pixel block of size N×N in the current frame, and then use an M×M search window to select a pixel block in the reference frame. During the matching process, the most matching pixel block can be found in the search window according to the minimum mean square error matching rule; in order to ensure efficient and accurate matching, the second denoised image and the previous frame image after denoising can also be decomposed into a Gaussian pyramid, and then pixel block matching is performed on the second denoised image and the previous frame image after denoising at different resolutions based on the Gaussian pyramid to obtain the final pixel block matching result.

[0130] Exemplarily, the electronic device may convert the second denoised image and the previous denoised image into representations of different resolutions, the bottom of the Gaussian pyramid is a high-resolution representation of the second denoised image and the previous denoised image, and the top is a low-resolution approximation of the second denoised image and the previous denoised image. Then, starting from the top of the Gaussian pyramid, pixel blocks of the second denoised image with the lowest resolution and the previous denoised image are matched. Since the resolution is the lowest, the processing speed is faster, but the matching accuracy is lower. After obtaining the first matching result between the second denoised image with the lowest resolution and the previous denoised image, the electronic device may determine a search area for performing the next pixel block matching according to the first matching result. For example, the search area may be an area in the second denoised image with a higher resolution corresponding to the second layer of the Gaussian pyramid, thereby continuing to determine the second matching result at this resolution, and determining the search area of ​​the next layer according to the second matching result, until the pixel block matching result of the second denoised image represented by the high resolution at the bottom and the previous denoised image is obtained. Based on this layer-by-layer matching from low resolution to high resolution, the processing efficiency of pixel block matching can be improved.

[0131] In some embodiments of the present application, after aligning the second denoised image and the previous frame image after denoising using the pixel block matching result, the first pixel block and the second pixel block that match each other in the second denoised image and the previous frame image after denoising are subtracted according to the pixel block matching result to obtain initial difference data; then, the initial difference data can be smoothed to obtain image difference data; the specific method used for image smoothing is not limited in this application, for example, a Gaussian filter smoothing method can be used to perform image smoothing on the initial difference data to obtain image difference data.

[0132] In some embodiments of the present application, the electronic device may further perform morphological processing on the initial difference data after obtaining the initial difference data to expand the size of the motion area in the initial difference data, so that the processed difference data can contain more motion information and better avoid the "ghosting" problem; and then perform image smoothing processing on the processed difference data to obtain image difference data.

[0133] It should be noted that the image difference data can be understood as an image that can reflect the motion difference between frame images.

[0134] In some embodiments of the present application, a single-frame denoising neural network can be trained offline. After the training is completed, the trained network and denoising intensity parameters can be used to perform a first denoising process on the current frame image to obtain a first denoised image.

[0135] In some embodiments of the present application, the neural network used to perform the first denoising processing in the spatial domain may include modules such as convolution, downsampling, residual connection, and upsampling. Operators such as S2D and D2S may be added on the basis of these basic modules, and multiplication operators and addition operators may be set before the output layer to control the denoising results; wherein the multiplication operator can be used to perform the multiplication operation between the denoising intensity parameter and the denoising residual, and the addition operator can be used to perform the product of the denoising intensity parameter and the denoising residual, and the addition operation between the first image data corresponding to the input current frame image.

[0136] For example, Figure 4 As shown, the trained network may include an input layer and an output layer, and the input layer and the output layer may include multiple convolutional layers (Conv) and rectified linear unit layers (ReLU); the training purpose of the neural network is to train to obtain noise residuals, so that the trained network can first perform feature extraction processing on the current frame image to obtain first image data, and then multiply the noise residual by the noise reduction strength parameter to obtain a result back to the first image data to obtain a first denoised image; the above method for determining the first denoised image using the trained network can be as shown in the above formula (2); in this way, the noise reduction strength can be effectively controlled, thereby achieving a good noise reduction effect.

[0137] In some embodiments of the present application, during the process of training a neural network, the noise reduction intensity parameter used for training can be set to 1. After obtaining the trained network, the noise reduction intensity parameter corresponding to each frame image can be determined by the aforementioned formula (1), thereby utilizing the noise reduction intensity parameter and the trained network to implement the first noise reduction processing on the current frame image.

[0138] In some embodiments of the present application, the electronic device can determine the values ​​and parameters in the aforementioned formula (1) based on at least one of the sensitivity information, dynamic range information, scene detection results, and color temperature information of the current frame image, thereby being able to control different noise reduction intensities for different situations of the current frame image.

[0139] In some embodiments of the present application, the fusion parameters can be determined by the aforementioned formula (3), and then the first denoised image and the previous frame image after denoising are fused using the fusion parameters in the manner shown in the aforementioned formula (4) to obtain the current frame image after denoising, thereby determining the denoising result of the video based on the denoised current frame images corresponding to each current frame image in the video.

[0140] In some embodiments of the present application, the electronic device can also determine the numerical values ​​and parameters in the aforementioned formula (3) based on at least one of the sensitivity information, dynamic range information, scene detection results, and color temperature information of the current frame image, so as to determine different fusion parameters for different situations of the current frame image and obtain a better fusion effect.

[0141] In some embodiments of the present application, the electronic device may also determine local noise reduction parameters based on at least one of the brightness information, color information and face detection results of the current frame image, and then perform local noise reduction processing on the current frame image according to the local noise reduction parameters to obtain a third noise reduction image, and then perform fusion processing on the first noise reduction image, the third noise reduction image and the previous frame image after noise reduction to obtain the current frame image after noise reduction.

[0142] In summary, the embodiment of the present application combines a neural network spatial noise reduction scheme based on deep learning with a temporal noise reduction scheme to improve the image quality of the video after noise reduction. Compared with the current related noise reduction methods, the embodiment of the present application can better retain the texture while removing noise, and can reduce the "ghosting" and other problems of traditional temporal noise reduction; compared with the current related neural network scheme based on deep learning, the method of the embodiment of the present application has a better overall effect, higher adjustability, and can better integrate the information of the original current frame image.

[0143] An embodiment of the present application provides a video denoising method, in which an electronic device determines image difference data based on a current frame image of a video and a previous frame image after denoising; performs a first denoising process on the current frame image based on the image difference data to obtain a first denoised image; and performs a fusion process on the first denoised image and the previous frame image after denoising to obtain a current frame image after denoising. It can be seen that in the embodiments of the present application, when the electronic device performs noise reduction on the video, it can perform noise reduction on each current frame image in the video, thereby completing the noise reduction of the entire video, and when performing noise reduction on the current frame image, it can first obtain the previous frame image after noise reduction, determine the image difference data between the current frame image and the previous frame image after noise reduction, and use the image difference data to perform the first noise reduction processing on the current frame image, thereby realizing spatial noise reduction of the current frame image based on the image difference data; then, the obtained first noise reduction image and the previous frame image after noise reduction are fused in the time domain, which can realize a joint noise reduction scheme of spatial and temporal domains, so that the final noise reduction current frame image can better retain the texture while removing the noise, reduce the "ghost" problem that occurs during noise reduction, and finally obtain the noise reduction result of the video from the current frame images after noise reduction, which can greatly improve the noise reduction effect of the video noise reduction.

[0144] Based on the above embodiment, in another embodiment of the present application, Figure 5 Schematic diagram of the structure of the electronic device proposed in the embodiment of the present application Figure 1 ,like Figure 5 As shown, the electronic device 1 proposed in the embodiment of the present application may include: a determination unit 11, a noise reduction unit 12 and a fusion unit 13.

[0145] The determination unit 11 may be configured to determine image difference data based on a current frame image of a video and a previous frame image after noise reduction.

[0146] The noise reduction unit 12 may be configured to perform a first noise reduction process on the current frame image based on the image difference data to obtain a first noise reduction image.

[0147] The fusion unit 13 may be used to fuse the first denoised image and the previous denoised frame image to obtain a denoised current frame image.

[0148] In some embodiments of the present application, the denoising unit 12 can be used to determine a denoising intensity parameter based on image difference data; and perform a first denoising process on the current frame image based on the denoising intensity parameter and a trained network to obtain a first denoised image.

[0149] In some embodiments of the present application, the determination unit 11 may also be used to determine the noise reduction intensity parameter according to a first value when a pixel value of the image difference data is less than a first threshold; and to determine the noise reduction intensity parameter according to a second value when a pixel value of the image difference data is greater than or equal to the first threshold and less than or equal to a second threshold; and to determine the noise reduction intensity parameter according to a third value when the pixel value of the image difference data is greater than the second threshold.

[0150] In some embodiments of the present application, the fusion unit 13 can also be used to determine fusion parameters based on image difference data; and fuse the first denoised image and the previous denoised frame image based on the fusion parameters to obtain the denoised current frame image.

[0151] In some embodiments of the present application, the fusion unit 13 may also be used to determine the fusion parameter according to a fourth value when the pixel value of the image difference data is less than a third threshold; and to determine the fusion parameter according to a fifth value when the pixel value of the image difference data is greater than or equal to the third threshold and less than or equal to the fourth threshold; and to determine the fusion parameter according to a sixth value when the pixel value of the image difference data is greater than the fourth threshold.

[0152] In some embodiments of the present application, the determination unit 11 may also be configured to determine a second value according to the first preset parameter and a pixel value of the image difference data.

[0153] In some embodiments of the present application, the determination unit 11 may also be configured to determine a fifth value according to the second preset parameter and the pixel value of the image difference data.

[0154] In some embodiments of the present application, the determination unit 11 can also be used to determine a first threshold, a second threshold, a third threshold, a fourth threshold, a first preset parameter and a second preset parameter based on the first image parameter; wherein the first image parameter includes at least one of the sensitivity information, dynamic range information, scene detection results and color temperature information of the current frame image.

[0155] In some embodiments of the present application, the denoising unit 12 can also be used to perform feature extraction processing on the current frame image based on the feature extraction layer in the trained network to obtain first image data; and perform first denoising processing on the first image data based on the denoising intensity parameter and the noise residual in the trained network to obtain a first denoised image.

[0156] In some embodiments of the present application, the determination unit 11 can also be used to perform a second denoising process on the current frame image to obtain a second denoised image; and perform pixel block division processing on the previous frame image after denoising to obtain a division result; and perform pixel block matching processing on the division result and the second denoised image to obtain a pixel block matching result; and determine image difference data based on the pixel block matching result.

[0157] In some embodiments of the present application, the pixel block matching result is used to indicate the second pixel blocks in the second denoised image that match the first pixel blocks of each division in the division result; the determination unit 11 can also be used to determine the initial difference data based on the difference between the first pixel block and the second pixel block; and perform image smoothing on the initial difference data to obtain image difference data.

[0158] In some embodiments of the present application, the determination unit 11 can also be used to determine the local noise reduction parameter according to the second image parameter; wherein the second image parameter includes at least one of the brightness information, color information and face detection result of the current frame image.

[0159] In some embodiments of the present application, the noise reduction unit 12 may also be configured to perform local noise reduction processing on the current frame image according to the local noise reduction parameters to obtain a third noise reduction image.

[0160] In some embodiments of the present application, the fusion unit 13 may also be used to perform fusion processing on the first denoised image, the third denoised image and the previous frame image after denoising to obtain the current frame image after denoising.

[0161] In the embodiments of the present application, further, Figure 6 Schematic diagram of the structure of the electronic device proposed in the embodiment of the present application Figure 2 ,like Figure 6 As shown, the electronic device 1 proposed in the embodiment of the present application may also include a processor 14 and a memory 15 storing executable instructions of the processor 14; further, the electronic device 1 may also include a communication interface 16, and a bus 17 for connecting the processor 14, the memory 15 and the communication interface 16.

[0162] In the embodiment of the present application, the processor 14 can be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, and a microprocessor. It can be understood that for different devices, the electronic device used to implement the function of the processor can also be other, and the embodiment of the present application is not specifically limited. The electronic device 1 can also include a memory 15, which can be connected to the processor 14, wherein the memory 15 is used to store executable program code, the program code includes computer operation instructions, and the memory 15 may include a high-speed RAM memory, and may also include a non-volatile memory, for example, at least two disk memories.

[0163] In the embodiment of the present application, the bus 17 is used to connect the communication interface 16, the processor 14 and the memory 15, and the mutual communication between these devices.

[0164] In the embodiment of the present application, the memory 15 is used to store instructions and data.

[0165] Further, in an embodiment of the present application, the processor 14 is used to determine image difference data based on a current frame image of a video and a previous frame image after noise reduction;

[0166] Performing a first noise reduction process on the current frame image based on the image difference data to obtain a first noise reduction image;

[0167] The first denoised image and the denoised previous frame image are fused to obtain a denoised current frame image.

[0168] In practical applications, the memory 15 may be a volatile memory, such as a random access memory (RAM); or a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk (HDD) or a solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 14.

[0169] In addition, each functional module in this embodiment can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or software functional modules.

[0170] If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method of this embodiment. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0171] An embodiment of the present application provides an electronic device, which determines image difference data based on a current frame image of a video and a previous frame image after noise reduction; performs a first noise reduction process on the current frame image based on the image difference data to obtain a first noise reduction image; and performs a fusion process on the first noise reduction image and the previous frame image after noise reduction to obtain a current frame image after noise reduction. It can be seen that in the embodiments of the present application, when the electronic device performs noise reduction on the video, it can perform noise reduction on each current frame image in the video, thereby completing the noise reduction of the entire video, and when performing noise reduction on the current frame image, it can first obtain the previous frame image after noise reduction, determine the image difference data between the current frame image and the previous frame image after noise reduction, and use the image difference data to perform the first noise reduction processing on the current frame image, thereby realizing spatial noise reduction of the current frame image based on the image difference data; then, the obtained first noise reduction image and the previous frame image after noise reduction are fused in the time domain, which can realize a joint noise reduction scheme of spatial and temporal domains, so that the final noise reduction current frame image can better retain the texture while removing the noise, reduce the "ghost" problem that occurs during noise reduction, and finally obtain the noise reduction result of the video from the current frame images after noise reduction, which can greatly improve the noise reduction effect of the video noise reduction.

[0172] Specifically, the program instructions corresponding to a video noise reduction method in this embodiment can be stored in a storage medium such as a CD, a hard disk, a USB flash drive, etc. When the program instructions corresponding to a video noise reduction method in the storage medium are read or executed by an electronic device, the following steps are included:

[0173] Determine image difference data based on a current frame image of the video and a previous frame image after noise reduction;

[0174] Performing a first noise reduction process on the current frame image based on the image difference data to obtain a first noise reduction image;

[0175] The first denoised image and the denoised previous frame image are fused to obtain a denoised current frame image.

[0176] An embodiment of the present application provides a computer program product, including a computer program or instructions. When the computer program or instructions are executed by a processor, the computer executes the steps in the method provided in the above method embodiment.

[0177] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of hardware embodiments, software embodiments, or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0178] The present application is described with reference to implementation flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0179] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which is implemented in the implementation flow diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0180] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing the steps in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0181] The above are only preferred embodiments of the present application and are not intended to limit the protection scope of the present application.

Claims

1. A video noise reduction method, characterized in that: The method comprises: Determine image difference data based on a current frame image of the video and a previous frame image after noise reduction; Performing a first noise reduction process on the current frame image based on the image difference data to obtain a first noise reduction image; The first denoised image and the denoised previous frame image are fused to obtain a denoised current frame image.

2. The video noise reduction method according to claim 1, characterized in that: The performing a first noise reduction process on the current frame image based on the image difference data to obtain a first noise reduction image includes: determining a noise reduction intensity parameter based on the image difference data; A first denoising process is performed on the current frame image based on the denoising intensity parameter and the trained network to obtain a first denoised image.

3. The video noise reduction method according to claim 2, characterized in that: The determining of the noise reduction intensity parameter based on the image difference data comprises: When the pixel value of the image difference data is less than a first threshold, determining the noise reduction intensity parameter according to a first value; When the pixel value of the image difference data is greater than or equal to the first threshold value and less than or equal to a second threshold value, determining the noise reduction intensity parameter according to a second value; wherein the first threshold value is less than the second threshold value; When the pixel value of the image difference data is greater than the second threshold, the noise reduction intensity parameter is determined according to a third value.

4. The video noise reduction method according to claim 1, characterized in that: The step of fusing the first denoised image with the previous denoised frame image to obtain a denoised current frame image includes: determining a fusion parameter based on the image difference data; The first denoised image and the denoised previous frame image are fused based on the fusion parameters to obtain the denoised current frame image.

5. The video noise reduction method according to claim 4, characterized in that: The determining of fusion parameters based on the image difference data comprises: When the pixel value of the image difference data is less than a third threshold, determining the fusion parameter according to a fourth value; When the pixel value of the image difference data is greater than or equal to the third threshold value and less than or equal to the fourth threshold value, determining the fusion parameter according to a fifth value; wherein the third threshold value is less than the fourth threshold value; When the pixel value of the image difference data is greater than the fourth threshold, the fusion parameter is determined according to a sixth value.

6. The video noise reduction method according to claim 3, characterized in that: The method further comprises: The second value is determined according to the first preset parameter and the pixel value of the image difference data.

7. The video noise reduction method according to claim 5, characterized in that: The method further comprises: The fifth value is determined according to a second preset parameter and a pixel value of the image difference data.

8. The video noise reduction method according to any one of claims 3, 5 to 7, characterized in that: The method further comprises: A first threshold, a second threshold, a third threshold, a fourth threshold, a first preset parameter and a second preset parameter are determined according to a first image parameter; wherein the first image parameter includes at least one of sensitivity information, dynamic range information, scene detection results and color temperature information of the current frame image.

9. The video noise reduction method according to claim 2, characterized in that: The performing a first denoising process on the current frame image based on the denoising intensity parameter and the trained network to obtain a first denoised image includes: Performing feature extraction processing on the current frame image based on the feature extraction layer in the trained network to obtain first image data; The first denoising process is performed on the first image data based on the denoising intensity parameter and the noise residual in the trained network to obtain the first denoised image.

10. The video noise reduction method according to claim 1, characterized in that: The determining of image difference data based on the current frame image of the video and the previous frame image after noise reduction includes: Performing a second noise reduction process on the current frame image to obtain a second noise reduction image; Perform pixel block division processing on the previous frame image after noise reduction to obtain a division result; Performing pixel block matching processing on the division result and the second denoised image to obtain a pixel block matching result; The image difference data is determined based on the pixel block matching result.

11. The video noise reduction method according to claim 10, characterized in that: The pixel block matching result is used to indicate the second pixel blocks in the second denoised image that are respectively matched with each divided first pixel block in the division result; The determining the image difference data based on the pixel block matching result comprises: determining initial difference data according to a difference between the first pixel block and the second pixel block; Performing image smoothing processing on the initial difference data to obtain the image difference data.

12. The video noise reduction method according to claim 1, characterized in that: The method further comprises: Determining a local noise reduction parameter according to a second image parameter; wherein the second image parameter includes at least one of brightness information, color information, and a face detection result of the current frame image; Performing local noise reduction processing on the current frame image according to the local noise reduction parameters to obtain a third noise reduction image; The first denoised image, the third denoised image and the denoised previous frame image are fused to obtain the denoised current frame image.

13. An electronic device, characterized in that: The electronic device includes a determination unit, a noise reduction unit and a fusion unit; The determining unit is used to determine image difference data based on a current frame image of the video and a previous frame image after noise reduction; The denoising unit is configured to perform a first denoising process on the current frame image based on the image difference data to obtain a first denoised image; The fusion unit is used to fuse the first denoised image and the previous denoised frame image to obtain a denoised current frame image.

14. An electronic device, characterized in that: The electronic device comprises a processor and a memory storing instructions executable by the processor; when the instructions are executed by the processor, the method according to any one of claims 1 to 12 is implemented.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

16. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps in the method according to any one of claims 1 to 12 are implemented.

Images