Video Image Denoising Method and Apparatus Based on Temporal Information and Weighting Factors

Abstract

This invention relates to the field of video image denoising technology, and discloses a video image denoising method and apparatus based on temporal information and weighting factors. The method includes: determining the optimal reference image block for an image block to be processed based on weighting factors; determining the motion intensity of the image block to be processed based on temporal information of the image block to be processed and the optimal reference image block; determining the reference block filtering coefficients of the optimal reference image block based on the motion intensity; calculating the reference pixel filtering coefficients and the pixel filtering coefficients to be processed for reference pixels based on the reference block filtering coefficients; and performing denoising on the pixel to be processed based on the pixel filtering coefficients and the reference pixel filtering coefficients. Therefore, implementing this invention can determine the optimal reference image block based on a local weighting factor strategy, which can improve the efficiency and accuracy of image and video denoising and reduce the probability of inter-frame flicker and motion blur problems after denoising.

Description

Technical Field

[0001] This invention relates to the field of video image denoising technology, and in particular to a video image denoising method and apparatus based on temporal information and weighting factors. Background Technology

[0002] With the rapid development of Internet technology, the demand for ultra-high-definition video is becoming increasingly strong. Noise is inevitably introduced during the video acquisition process, and noise reduction technology has always been one of the key technologies of concern in the fields of video encoding and image processing.

[0003] Currently, noise reduction methods are mainly divided into two categories: traditional noise reduction methods and learning-based noise reduction methods. Among them, a commonly used method in traditional video coding noise reduction is spatiotemporal motion adaptive noise reduction (3D noise reduction). This method utilizes spatial similarity and the temporal continuity between consecutive frames and the current frame to remove noise from the video. Compared to spatial noise reduction (2D noise reduction), 3D noise reduction can reduce background flicker, and motion adaptation can reduce the "ghosting" problem that occurs in temporal noise reduction. However, when continuous background areas appear in the video sequence, inter-frame flicker and motion ghosting are likely to occur after noise reduction. Learning-based noise reduction methods use network models for motion estimation, noise level estimation, and noise reduction intensity determination. The network models require a large number of parameters to train, making real-time noise reduction difficult and hindering its application in real-time video coding. Therefore, proposing a technical solution that can reduce the probability of inter-frame flicker and motion ghosting after video noise reduction is particularly important. Summary of the Invention

[0004] This invention provides a video image denoising method and apparatus based on temporal information and weighting factors, which can help reduce the probability of inter-frame flickering and motion blur problems after video denoising.

[0005] To address the aforementioned technical problems, the first aspect of this invention discloses a video image noise reduction method based on temporal information and weighting factors, the method comprising:

[0006] At least one image block to be processed is obtained from the image frame to be processed. For each image block to be processed, the best reference image block corresponding to the image block to be processed is determined in the reference image frame corresponding to the image frame to be processed according to a preset weight factor.

[0007] For each image block to be processed, the temporal information between the image block to be processed and the corresponding best reference image block is calculated, and the temporal information includes similarity and motion vectors;

[0008] For each image block to be processed, the motion intensity of the image block to be processed is determined according to the temporal information corresponding to the image block to be processed, and the reference block filtering coefficient of the best reference image block corresponding to the image block to be processed is determined according to the motion intensity.

[0009] For each image block to be processed, the reference pixel filtering coefficient of each reference pixel in the best reference image block is calculated based on the reference block filtering coefficient of the best reference image block corresponding to the image block to be processed, and the filter coefficient of each pixel to be processed in the image block to be processed is calculated based on the reference pixel filtering coefficient of each reference pixel.

[0010] For each image block to be processed, for each pixel to be processed in the image block to be processed, the pixel to be processed is filtered and denoised according to the pixel to be processed filter coefficient and the reference pixel filter coefficient of the corresponding reference pixel to obtain the denoised output result corresponding to the pixel to be processed.

[0011] A second aspect of the present invention discloses a video image noise reduction device based on temporal information and weighting factors, the device comprising:

[0012] The acquisition module is used to acquire at least one image block to be processed in the image frame to be processed. For each image block to be processed, the best reference image block corresponding to the image block to be processed is determined in the reference image frame corresponding to the image frame to be processed according to a preset weight factor.

[0013] The calculation module is used to calculate, for each image block to be processed, the temporal information between the image block to be processed and the corresponding best reference image block, wherein the temporal information includes similarity and motion vector;

[0014] The determination module is used to determine the motion intensity of each image block to be processed based on the temporal information corresponding to the image block to be processed, and to determine the reference block filtering coefficient of the best reference image block corresponding to the image block to be processed based on the motion intensity.

[0015] The calculation module is further configured to, for each image block to be processed, calculate the reference pixel filtering coefficient of each reference pixel in the best reference image block according to the reference block filtering coefficient of the best reference image block corresponding to the image block to be processed, and calculate the pixel filtering coefficient of each pixel to be processed in the image block to be processed according to the reference pixel filtering coefficient of each reference pixel.

[0016] The noise reduction module is used to perform noise reduction on each pixel to be processed in each image block to be processed, based on the pixel to be processed filtering coefficient and the reference pixel filtering coefficient of the reference pixel corresponding to the pixel to be processed, to obtain the noise reduction output result corresponding to the pixel to be processed.

[0017] A third aspect of the present invention discloses another video image noise reduction device based on temporal information and weighting factors, the device comprising:

[0018] Memory containing executable program code;

[0019] A processor coupled to the memory;

[0020] The processor calls the executable program code stored in the memory to execute the video image noise reduction method based on temporal information and weighting factors disclosed in the first aspect of the present invention.

[0021] The fourth aspect of the present invention discloses a computer storage medium storing computer instructions, which, when invoked, are used to execute the video image noise reduction method based on temporal information and weighting factors disclosed in the first aspect of the present invention.

[0022] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0023] In this embodiment of the invention, the optimal reference image block corresponding to the image block to be processed in the image frame to be processed is determined according to a preset weighting factor. Based on the temporal information between the image block to be processed and the corresponding optimal reference image block, the motion intensity of the image block to be processed is determined. Based on the motion intensity, the reference block filtering coefficient of the optimal reference image block corresponding to the image block to be processed is determined. Based on the reference block filtering coefficient, the reference pixel filtering coefficient of each reference pixel in the optimal reference image block and the pixel filtering coefficient of each pixel to be processed in the image block to be processed are calculated. The pixel to be processed is then filtered and denoised based on the pixel filtering coefficient and the reference pixel filtering coefficient to obtain the denoised output result corresponding to the pixel to be processed. It can be seen that implementing this invention can determine the optimal reference image block corresponding to the image block to be processed based on a local weighting factor strategy, improving the accuracy of determining the optimal reference image block. Calculating temporal information on a block-by-block basis reduces the occupation of additional cached data with fewer reference frames, thereby improving the efficiency and accuracy of image and video filtering and denoising, and reducing the probability of inter-frame flickering and motion blur problems after video denoising. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart illustrating a video image noise reduction method based on temporal information and weighting factors disclosed in an embodiment of the present invention.

[0026] Figure 2 This is a flowchart illustrating another video image noise reduction method based on temporal information and weighting factors disclosed in an embodiment of the present invention;

[0027] Figure 3 This is a schematic diagram of the structure of a video image noise reduction device based on temporal information and weighting factors disclosed in an embodiment of the present invention;

[0028] Figure 4 This is a schematic diagram of another video image noise reduction device based on temporal information and weighting factors disclosed in an embodiment of the present invention. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or end that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.

[0031] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0032] This invention discloses a video image denoising method and apparatus based on temporal information and weighting factors. It can determine the optimal reference image block corresponding to the image block to be processed based on a local weighting factor strategy, improving the accuracy of determining the optimal reference image block. By calculating temporal information on a block-by-block basis, fewer reference frames reduce the occupation of additional buffered data, thereby improving the efficiency and accuracy of image and video filtering and denoising, and reducing the probability of inter-frame flickering and motion blur problems after video denoising. Detailed descriptions follow.

[0033] Example 1

[0034] Please see Figure 1 , Figure 1 This is a flowchart illustrating a video image denoising method based on temporal information and weighting factors disclosed in an embodiment of the present invention. Figure 1 The described video image denoising method based on temporal information and weighting factors can be applied to a video image denoising device based on temporal information and weighting factors. This device may include an intelligent server or intelligent platform for denoising video images. The intelligent server may be a local server or a cloud server; this embodiment of the invention does not impose limitations. Figure 1 As shown, this video image denoising method based on temporal information and weighting factors can include the following operations:

[0035] 101. Obtain at least one image block to be processed in the image frame to be processed. For each image block to be processed, determine the best reference image block corresponding to the image block to be processed in the reference image frame corresponding to the image frame to be processed according to the preset weight factor.

[0036] In this embodiment of the invention, optionally, the image frame to be processed can be divided into at least one macroblock, i.e., image block to be processed. Each image block to be processed is of equal size. Each image frame to be processed corresponds to a reference image frame. Each image block to be processed in each image frame corresponds to an optimal reference image block in the reference image frame. The optimal reference image block corresponding to the image block to be processed can be determined according to a preset weight factor. Specifically, the optimal reference image block corresponding to the image block to be processed can be determined based on a strategy of local weight factor.

[0037] 102. For each image block to be processed, calculate the temporal information between the image block to be processed and the corresponding best reference image block. Based on the temporal information corresponding to the image block to be processed, determine the motion intensity of the image block to be processed. Based on the motion intensity, determine the reference block filtering coefficient of the best reference image block corresponding to the image block to be processed.

[0038] In this embodiment of the invention, optionally, the temporal information between the image block to be processed and the corresponding best reference image block may include the similarity and motion vector between the image block to be processed and the corresponding best reference image block. The motion intensity of the image block to be processed can be determined based on the similarity and motion vector between the image block to be processed and the corresponding best reference image block, and the reference block filter coefficient of the best reference image block corresponding to the image block to be processed can be selected from a preset filter coefficient table based on the motion intensity.

[0039] 103. For each image block to be processed, calculate the reference pixel filtering coefficient of each reference pixel in the best reference image block based on the reference block filtering coefficient of the best reference image block corresponding to the image block to be processed, and calculate the pixel filtering coefficient of each pixel to be processed in the image block to be processed based on the reference pixel filtering coefficient of each reference pixel.

[0040] 104. For each image block to be processed, for each pixel to be processed in the image block to be processed, filter and denoise the pixel to be processed according to the pixel to be processed filter coefficient and the reference pixel filter coefficient of the corresponding reference pixel to obtain the denoising output result corresponding to the pixel to be processed.

[0041] In this embodiment of the invention, optionally, for each image block to be processed, the pixel to be processed can be filtered and denoised according to the filtering coefficient of the pixel to be processed and the filtering coefficient of the reference pixel to obtain the denoised output result corresponding to the pixel to be processed. Then, the denoised output results of all pixels of the image block to be processed are combined into a new image block to replace the image block to be processed, thereby completing the filtering and denoising of the image block to be processed. This invention does not limit the scope of the invention.

[0042] It is evident that implementation Figure 1The described video image denoising method based on temporal information and weighting factors can determine the optimal reference image block corresponding to the image block to be processed in the image frame to be processed according to the preset weighting factors. Based on the temporal information between the image block to be processed and the corresponding optimal reference image block, the motion intensity of the image block to be processed is determined. Based on the motion intensity, the reference block filtering coefficient of the optimal reference image block corresponding to the image block to be processed is determined. Based on the reference block filtering coefficient, the reference pixel filtering coefficient of each reference pixel in the optimal reference image block and the pixel filtering coefficient of each pixel to be processed in the image block to be processed are calculated. The pixel to be processed is then filtered and denoised based on the pixel filtering coefficient and the reference pixel filtering coefficient to obtain the denoised output result corresponding to the pixel to be processed. This method can determine the optimal reference image block corresponding to the image block to be processed based on a local weighting factor strategy, improving the accuracy of determining the optimal reference image block. Calculating temporal information on a block-by-block basis reduces the amount of additional buffered data required by fewer reference frames, thereby improving the efficiency and accuracy of image and video filtering and denoising, and reducing the probability of inter-frame flicker and motion blur problems after video denoising.

[0043] In an optional embodiment, for each image block to be processed, calculating the reference pixel filtering coefficient of each reference pixel in the best reference image block based on the reference block filtering coefficient of the best reference image block corresponding to the image block to be processed may include the following operations:

[0044] For each image block to be processed, for each pixel to be processed in the image block to be processed, calculate the absolute value of the difference between the pixel to be processed and each corresponding reference pixel in the best reference image block;

[0045] For each image block to be processed, the extreme value of the absolute value of the difference between the image block to be processed and the corresponding best reference image block is determined in each absolute value of the difference, and the difference between the extreme values ​​of the absolute values ​​of the difference is calculated based on the extreme values ​​of the absolute values ​​of the difference. The extreme values ​​of the absolute values ​​of the difference include the maximum value of the absolute value of the difference and the minimum value of the absolute value of the difference.

[0046] For each image block to be processed, for each pixel to be processed in the image block to be processed, the normalization result of the absolute value of the difference between the pixel to be processed and the corresponding reference pixel within the preset target range is calculated based on the absolute value of the difference between the pixel to be processed and the corresponding reference pixel, the minimum absolute value of the difference, and the extreme value difference of the absolute value of the difference.

[0047] For each image block to be processed, for each pixel to be processed in the image block to be processed, the scaling factor corresponding to the pixel to be processed is calculated based on the normalization result between the pixel to be processed and the corresponding reference pixel. Then, based on the scaling factor and the reference block filtering factor of the best reference image block corresponding to the image block to be processed, the reference pixel filtering factor of each reference pixel in the best reference image block corresponding to the image block to be processed is calculated.

[0048] In this optional embodiment, the formula for calculating the absolute value of the difference between the pixel to be processed and each corresponding reference pixel in the best reference image block is:

[0049] pixeldiffval k (i,j)=abs(curpixelval k (i,j)-refpixelval k (i,j))

[0050] Among them, pixeldiffval k (i,j) represents the absolute value of the difference between the pixel point to be processed with coordinates (i,j) in the kth image block to be processed and the corresponding reference pixel point;

[0051] Optionally, the extreme values ​​of the absolute differences between the image patch to be processed and the corresponding best reference image patch are determined in each absolute difference. The extreme values ​​of the absolute differences may include the maximum and minimum absolute values ​​of the differences, and the specific formulas include:

[0052] maxpixeldiff k =max(pixeldiffval) k (i,j)), minpixeldiff k =min(pixeldiffval) k (i,j))

[0053] Among them, maxpixeldiff k minpixeldiff represents the maximum absolute value of the difference corresponding to the k-th image patch to be processed. k This represents the minimum absolute value of the difference corresponding to the k-th image block to be processed;

[0054] Optionally, the difference between the absolute values ​​and extreme values ​​can be calculated based on the extreme values ​​of the difference. Specific calculation formulas include:

[0055] gap k =maxpixeldiff k -minpixeldiff k

[0056] Among them, gap k This represents the absolute value and extreme value difference corresponding to the k-th image block to be processed.

[0057] In this optional embodiment, for each image block to be processed, for each pixel to be processed in the image block to be processed, the normalization result of the absolute value of the difference between the pixel to be processed and the corresponding reference pixel within a preset target range is calculated based on the absolute value of the difference between the pixel to be processed and the corresponding reference pixel, the minimum absolute value of the difference, and the extreme difference of the absolute value of the difference. The preset target range can be set by the user. Based on the normalization result between the pixel to be processed and the corresponding reference pixel, the scaling factor corresponding to the pixel to be processed is calculated. Based on the scaling factor and the reference block filtering factor of the best reference image block corresponding to the image block to be processed, the reference pixel filtering factor of each reference pixel in the best reference image block corresponding to the image block to be processed is calculated. This embodiment does not limit this.

[0058] As can be seen, implementing this optional embodiment can calculate the absolute value of the difference between the pixel to be processed and each corresponding reference pixel in the best reference image block, and determine the extreme value and the difference between the extreme values ​​of the absolute values ​​of the difference. Based on the absolute value of the difference, the minimum value of the absolute value of the difference, and the difference between the extreme values ​​of the absolute values ​​of the difference, the normalized result of the absolute value of the difference within a preset target range is calculated. Based on the normalized result between the pixel to be processed and the corresponding reference pixel, the scaling factor corresponding to the pixel to be processed is calculated. Based on the scaling factor and the reference block filtering factor of the best reference image block corresponding to the image block to be processed, the reference pixel filtering factor of each reference pixel in the best reference image block corresponding to the image block to be processed is calculated. It can calculate the scaling factor of the exponential model based on the absolute value of the difference of the image block to be processed, and calculate the reference pixel filtering factor of each reference pixel based on the scaling factor, thereby improving the accuracy of the calculated reference pixel filtering factor, and thus improving the accuracy and efficiency of filtering and denoising the image block to be processed, and reducing the probability of inter-frame flickering and motion blur problems after video denoising.

[0059] In another optional embodiment, the formula for calculating the normalization result includes:

[0060]

[0061] Among them, normalpixeldiffval k (i,j) represents the normalized result of the absolute value of the difference between the pixel at coordinates (i,j) in the k-th image block of the image frame and the corresponding reference pixel in the range [-a, a]. k(i,j) represents the absolute value of the difference between the pixel at coordinates (i,j) in the k-th image block and the corresponding reference pixel, minpixeldiff. k The gap represents the minimum absolute value of the difference corresponding to the k-th image patch to be processed. k This represents the absolute value and extreme value difference of the difference corresponding to the k-th image block to be processed;

[0062] Furthermore, the formula for calculating the filter coefficients of each reference pixel in the best reference image block corresponding to each image block to be processed includes:

[0063] pixelRefCoef k (i,j)=refblkCoef k *exp(-normalpixeldiffval k (i,j))

[0064] Among them, pixelRefCoef k (i,j) represents the reference pixel filtering coefficient corresponding to the reference pixel at coordinates (i,j) in the k-th image block to be processed. refblkCoef k The reference block filter coefficients of the best reference image block corresponding to the k-th image block to be processed are exp(-normalpixeldiffval). k (i,j)) represents the scaling factor corresponding to the pixel point with coordinates (i,j) in the k-th image block to be processed;

[0065] Furthermore, the formula for calculating the filter coefficients of each pixel in each image block to be processed includes:

[0066] pixelCurCoef k (i,j)=2 W -pixelRefCoef k (i,j)

[0067] Among them, pixelCurCoef k (i,j) represents the filtering coefficient of the pixel to be processed with coordinates (i,j) in the k-th image block to be processed, and W represents the preset weight value precision.

[0068] In this optional embodiment, optionally, normalpixeldiffval k(i,j) represents the normalized result of the absolute value of the difference between the pixel at coordinates (i,j) in the k-th image block of the image frame and the corresponding reference pixel in the range [-a, a]. The preset target range [-a, a] can be user-defined. For example, the preset target range [-a, a] can be [-0.5, 0.5]. k (i,j) represents the reference pixel filtering coefficient corresponding to the reference pixel at coordinates (i,j) in the k-th image block to be processed. refblkCoef k The reference block filter coefficients of the best reference image block corresponding to the k-th image block to be processed are exp(-normalpixeldiffval). k (i,j)) represents the scaling factor corresponding to the pixel with coordinates (i,j) in the k-th image block to be processed. pixelCurCoef k (i,j) represents the filtering coefficient of the pixel to be processed with coordinates (i,j) in the k-th image block to be processed, and W represents the preset weight value precision.

[0069] As can be seen, implementing this optional embodiment can calculate the normalized result of the absolute value of the difference within a preset target range based on the absolute value of the difference, the minimum absolute value of the difference, and the difference between the extreme values ​​of the absolute values ​​of the difference. Then, it can calculate the scaling factor of the exponential model based on the normalized result, and calculate the filter coefficient of the reference pixel and the filter coefficient of the pixel to be processed based on the scaling factor. This improves the accuracy and efficiency of the calculation of the pixel filter coefficient, thereby improving the accuracy and efficiency of filtering and denoising the image block to be processed, and reducing the probability of inter-frame flickering and motion blur problems after video denoising.

[0070] In another optional embodiment, for each image block to be processed, for each pixel to be processed in the image block to be processed, filtering and noise reduction are performed on the pixel to be processed according to the pixel to be processed filtering coefficient and the reference pixel filtering coefficient of the reference pixel corresponding to the pixel to be processed, and the noise reduction output result corresponding to the pixel to be processed may include the following operations:

[0071] For each image block to be processed, for each pixel to be processed in the image block to be processed, the pixel to be processed is filtered and denoised according to the pixel to be processed filter coefficient, the reference pixel filter coefficient of the reference pixel corresponding to the pixel to be processed and the preset filtering and denoising formula, so as to obtain the denoising output result corresponding to the pixel to be processed.

[0072] The filtering and noise reduction formulas include:

[0073]

[0074] denoiseOut k (i,j)=clip(0,255,denoiseOut k (i,j))

[0075] Among them, denoiseOut k (i,j) represents the noise reduction output of the pixel at coordinates (i,j) in the k-th image block of the image frame to be processed. (curpixelval) k (i,j) represents the pixel value of the pixel at coordinates (i,j) in the k-th image block to be processed, refpixelval k (i,j) represents the pixel value of the reference pixel corresponding to the pixel at coordinates (i,j) in the k-th image block to be processed. clip(0,255,denoiseOut) k (i,j)) means that the noise reduction output of the pixel with coordinates (i,j) in the k-th image block of the image frame to be processed is clamped within [0,255].

[0076] In this optional embodiment, optionally, denoiseOut k (i,j) represents the denoising output of the pixel at coordinates (i,j) in the k-th image block of the image frame to be processed. W represents the preset weight precision; the larger W is, the higher the precision. For example, W can be 4 or 6, but this embodiment does not limit it. clip(0,255,denoiseOut) k (i,j)) means that the noise reduction output of the pixel at coordinate (i,j) in the k-th image block to be processed in the image frame to be processed is clamped within [0,255]. For each image block to be processed, after obtaining the noise reduction output corresponding to each pixel in the image block to be processed, the noise reduction output of all pixels in the image block to be processed is combined into a new image block to replace the image block to be processed, thus completing the filtering and noise reduction of the image block to be processed.

[0077] As can be seen, implementing this optional embodiment can filter and denoise the pixel to be processed according to the preset filtering and denoising formula, obtain the denoising output result corresponding to the pixel to be processed, and clamp the denoising output result within [0,255]. The denoising output results of all pixels of the image block to be processed are combined into a new image block to replace the image block to be processed, thus completing the filtering and denoising of the image block to be processed. This can improve the efficiency and accuracy of filtering and denoising of images and videos, and reduce the probability of inter-frame flickering and motion blur problems after video denoising.

[0078] Example 2

[0079] Please see Figure 2 , Figure 2 This is a flowchart illustrating a video image denoising method based on temporal information and weighting factors disclosed in an embodiment of the present invention. Figure 2 The described video image denoising method based on temporal information and weighting factors can be applied to a video image denoising device based on temporal information and weighting factors. This device may include an intelligent server or intelligent platform for denoising video images. The intelligent server may be a local server or a cloud server; this embodiment of the invention does not impose limitations. Figure 2 As shown, this video image denoising method based on temporal information and weighting factors can include the following operations:

[0080] 201. Obtain at least one image block to be processed in the image frame to be processed, divide the image frame to be processed and the reference image frame corresponding to the image frame to be processed into regions, and obtain at least one region to be processed corresponding to the image frame to be processed and at least one reference region corresponding to the reference image frame.

[0081] In this embodiment of the invention, optionally, the reference image frame corresponding to the image frame to be processed may include at least the previous frame image of the image frame to be processed. That is, the reference image frame may include the previous frame image of the image frame to be processed, or it may include multiple previous frames of the image frame to be processed. After dividing the image frame to be processed and the reference image frame into regions, the size and number of at least one processing region corresponding to the image frame to be processed and at least one reference region corresponding to the reference image frame are the same. That is, the processing region and the reference region correspond one-to-one, and the region size of the processing region is larger than the image block size of the image block to be processed. That is, a processing region may include at least one image block to be processed. This invention does not impose any limitations.

[0082] 202. Statistically analyze the motion vector histogram of each reference region in the reference image frame. For each region to be processed, predict the motion / static type of the region to be processed based on the motion vector histogram of the reference region corresponding to the region to be processed, and determine the weight factor of the region to be processed based on the motion / static type of the region to be processed.

[0083] In this embodiment of the invention, optionally, the motion vector histogram of each reference region in the reference image frame can be statistically analyzed. For each region to be processed, the motion / static type of the region to be processed can be predicted based on the motion vector histogram of the reference region corresponding to the region to be processed. Further optionally, the pixel difference between the current block and the corresponding reference block at the same position in the previous frame can be statistically analyzed as a metric. This invention does not impose any limitations.

[0084] In this embodiment of the invention, optionally, the motion / static type of the region to be processed can include either a motion type or a static type. Due to noise, the similarity between the reference block and the current block is relatively high. Using the original Lambda will cause slight fluctuations in the motion vector of the background region. To reduce the impact of noise on motion estimation of the background region, this invention proposes a local Lambda strategy. The previous frame is divided into multiple regions, and the motion vector histogram of each region is calculated. The Lambda is adjusted according to the motion characteristics of each region. If the noise intensity of the current frame is relatively high, and the region where the current block is located is a static region, then the Lambda of the current region is relatively large. If the region where the current block is located is a moving region, then the Lambda of the current region remains unchanged. Specifically, the optimal reference image block corresponding to the image block to be processed can be determined based on a local weight factor strategy, that is, different weight factors for the region to be processed are determined according to the motion / static type of the region to be processed. Specifically, optionally, determining the weight factors for the region to be processed based on the motion / static type of the region to be processed can specifically include:

[0085] When the static / dynamic type of the region to be processed includes motion, the weighting factor of the region to be processed is calculated according to the first formula, wherein the first formula includes:

[0086]

[0087] When the static type of the region to be processed includes the motion type, the weight factor of the region to be processed is calculated according to the first formula, wherein the second formula includes:

[0088]

[0089] Wherein, LambdaOri represents the weight factor corresponding to the region to be processed in motion, deltaOri and alphaOri represent the auxiliary weight factors corresponding to the region to be processed in motion, LambdaOpt represents the weight factor corresponding to the region to be processed in stillness, deltaOpt and alphaOpt represent the auxiliary weight factors corresponding to the region to be processed in stillness, and qp represents the preset quantization coefficient.

[0090] In this embodiment of the invention, optionally, for example, when the static / dynamic type of the area to be processed includes the motion type, deltaOri = 13.7122, alphaOri = 4.2005; when the static type of the area to be processed includes the motion type, deltaOpt = 13.5, alphaOpt = 4.2. This invention does not impose any limitations.

[0091] 203. For each image block to be processed, calculate the cost between the image block to be processed and each reference image block in the corresponding reference image frame according to the weight factor of the region to be processed corresponding to the image block to be processed, and determine the reference image block corresponding to the cost that satisfies the preset selection conditions as the best reference image block corresponding to the image block to be processed.

[0092] In this embodiment of the invention, optionally, the cost that satisfies the preset selection conditions may include the minimum cost among each reference image block in the corresponding reference image frame, that is, the reference image block with the minimum cost is determined as the best reference image block corresponding to the image block to be processed. This invention does not limit this.

[0093] 204. For each image block to be processed, calculate the temporal information between the image block to be processed and the corresponding best reference image block. Based on the temporal information corresponding to the image block to be processed, determine the motion intensity of the image block to be processed. Based on the motion intensity, determine the reference block filtering coefficient of the best reference image block corresponding to the image block to be processed.

[0094] 205. For each image block to be processed, calculate the reference pixel filtering coefficient of each reference pixel in the best reference image block based on the reference block filtering coefficient of the best reference image block corresponding to the image block to be processed, and calculate the pixel filtering coefficient of each pixel to be processed in the image block to be processed based on the reference pixel filtering coefficient of each reference pixel.

[0095] 206. For each image block to be processed, for each pixel to be processed in the image block to be processed, filter and denoise the pixel to be processed according to the pixel to be processed filter coefficient and the reference pixel filter coefficient of the corresponding reference pixel to obtain the denoising output result corresponding to the pixel to be processed.

[0096] In this embodiment of the invention, for other descriptions of steps 204-206, please refer to the detailed description of steps 102-104 in Embodiment 1. These descriptions will not be repeated in this embodiment of the invention.

[0097] It is evident that implementation Figure 1The described video image denoising method based on temporal information and weighting factors can divide the image frame to be processed and the reference image frame into regions, statistically analyze the motion vector histogram of each reference region in the reference image frame, and predict the motion / static type of the image frame to be processed based on the motion vector histogram, thus improving the efficiency of determining the motion / static type of the image frame to be processed. Based on the motion / static type of the image frame to be processed, a weighting factor is determined for that image frame, and the cost between the image frame to be processed and each reference image frame is calculated based on the weighting factor. The reference image frame with the cost that meets the preset selection criteria is determined as the optimal reference image frame for the image frame to be processed. This method can determine the optimal reference image frame for the image frame to be processed through a local weighting factor strategy, improving the accuracy and efficiency of determining the optimal reference image frame while also improving the denoising performance of the image frame to be processed. To improve accuracy and efficiency, the system determines the optimal reference image block corresponding to the image block to be processed in the image frame based on a preset weighting factor. Then, based on the temporal information between the image block to be processed and the corresponding optimal reference image block, it determines the motion intensity of the image block to be processed. Based on the motion intensity, it determines the reference block filtering coefficient of the optimal reference image block corresponding to the image block to be processed. Using the reference block filtering coefficient, it calculates the reference pixel filtering coefficient for each reference pixel in the optimal reference image block and the pixel filtering coefficient for each pixel to be processed in the image block to be processed. Finally, it performs filtering and noise reduction on the pixel to be processed based on the pixel filtering coefficient and the reference pixel filtering coefficient, obtaining the noise reduction output result corresponding to the pixel to be processed. This method can improve the efficiency and accuracy of image and video filtering and noise reduction, and reduce the probability of inter-frame flickering and motion blur problems after video noise reduction.

[0098] In an optional embodiment, for each image block to be processed, determining the motion intensity of the image block to be processed based on the temporal information corresponding to the image block to be processed may include the following operations:

[0099] Obtain the level thresholds corresponding to the time domain information. The level thresholds include similarity level thresholds and motion vector level thresholds.

[0100] For each image block to be processed, the motion intensity of the image block to be processed is determined based on the target relationship between the temporal information corresponding to the image block to be processed and the level threshold. The target relationship includes the first relationship between the similarity of the image block to be processed and the similarity level threshold and the second relationship between the motion vector of the image block to be processed and the motion vector level threshold.

[0101] The first relationship includes:

[0102]

[0103] The second relationship includes:

[0104]

[0105] The motion intensity of the image patch to be processed includes:

[0106]

[0107] Among them, sadStatus k This represents the first relation corresponding to the k-th image block in the image frame to be processed, sad k mvStatus represents the similarity between the k-th image patch to be processed and the best reference image patch in the image frame to be processed. k This represents the second relation corresponding to the k-th image block in the image frame to be processed, mv k moveStatus represents the motion vector between the k-th image patch to be processed and the best reference image patch in the image frame to be processed. k represents the motion intensity of the k-th image block in the image frame to be processed, sadTh0-sadTh3 represents the similarity level threshold, and mvTh0-mvTh3 represents the motion vector level threshold.

[0108] In this optional embodiment, the level threshold corresponding to the temporal information can be preset by the user according to the actual video image noise reduction requirements, or it can be automatically set by the system based on historical video image noise reduction records. The level threshold may include a similarity level threshold and a motion vector level threshold. The target relationship between the temporal information corresponding to the image block to be processed and the level threshold may include a first relationship and a second relationship. The first relationship may represent the magnitude relationship between the similarity of the image block to be processed and the similarity level threshold, and the second relationship may include the magnitude relationship between the motion vector of the image block to be processed and the motion vector level threshold. Specifically, the first relationship may include:

[0109]

[0110] The second relation can include:

[0111]

[0112] The motion intensity of the image patch to be processed can include:

[0113]

[0114] Among them, sadStatus k This represents the first relation corresponding to the k-th image block in the image frame to be processed, sad k mvStatus represents the similarity between the k-th image patch to be processed and the best reference image patch in the image frame to be processed. kThis represents the second relation corresponding to the k-th image block in the image frame to be processed, mv k moveStatus represents the motion vector between the k-th image patch to be processed and the best reference image patch in the image frame to be processed. k represents the motion intensity of the k-th image block in the image frame to be processed, sadTh0-sadTh3 represents the similarity level threshold, and mvTh0-mvTh3 represents the motion vector level threshold.

[0115] In this optional embodiment, the formula for calculating the similarity between the k-th image block to be processed and the best reference image block in the image frame to be processed is:

[0116]

[0117] Among them, curpixelval k (i,j) represents the pixel value of the pixel at coordinates (i,j) in the k-th image block to be processed, refpixelval k (i,j) represents the pixel value of the reference pixel corresponding to the pixel point with coordinates (i,j) in the k-th image block to be processed, and L and K represent the number of pixels in the horizontal and vertical directions in the k-th image block to be processed, respectively.

[0118] The formula for calculating the motion vector between the k-th image block to be processed and the best reference image block in the image frame to be processed is:

[0119] mv k =abs(mvx) + abs(mvy)

[0120] Where mvx and mvy represent the motion vector components of the k-th image patch to be processed in the x and y directions, respectively.

[0121] In this optional embodiment, optionally, for each image block to be processed, the reference block filter coefficients of the best reference image block corresponding to the image block to be processed can be selected from a preset filter coefficient table according to the motion intensity of the image block to be processed. The filter coefficient table may include:

[0122] Filter coefficient table

[0123] <![CDATA[mvStatus k ]]> <![CDATA[refblkCoef k ]]> 0 14 1 10 2 8 3 6 4 0

[0124] Among them, refblkCoef kThis represents the reference block filtering coefficient of the best reference image block corresponding to the k-th image block to be processed. Optionally, refblkCoef decreases as the intensity of moveStatus increases. The data in the filtering coefficient table is only for illustrative purposes to show the correspondence between refblkCoef and moveStatus. This embodiment does not limit the specific values ​​of refblkCoef and moveStatus.

[0125] In this optional embodiment, the reference block filter coefficients refblkCoef of the optimal reference image block corresponding to the k-th image block to be processed are optionally determined. k Then, the filter coefficients of the k-th image block to be processed are (N-refblkCoef) k ), where N = 2 M M represents bit width.

[0126] As can be seen, implementing this optional embodiment can obtain the level threshold corresponding to the temporal information. The level threshold includes a similarity level threshold and a motion vector level threshold. Based on the target relationship between the temporal information corresponding to the image block to be processed and the level threshold, the motion intensity of the image block to be processed can be determined. It can determine the motion intensity of the image block to be processed based on the relationship between the similarity and motion vector corresponding to the image block to be processed and the corresponding threshold level, which can improve the accuracy and efficiency of determining the motion intensity of the image block to be processed, thereby improving the noise reduction efficiency of video images.

[0127] In another optional embodiment, for each image block to be processed, calculating the cost between the image block to be processed and each reference image block in the reference image frame based on the weight factor of the region to be processed corresponding to the image block to be processed may include the following operations:

[0128] For each image block to be processed, calculate the reference temporal information between the image block to be processed and each reference image block in the reference image frame;

[0129] For each image block to be processed, the cost between the image block to be processed and each reference image block in the reference image frame is calculated according to the weight factor of the region to be processed corresponding to the image block to be processed and the reference temporal information, according to the preset cost calculation formula.

[0130] The cost calculation formula includes:

[0131] cost k，p =sad k，p +mvbits k，p *Lambda k

[0132] Among them, cost k，pThis represents the cost between the k-th image block in the image frame to be processed and the p-th reference image block in the reference image frame. k，p The mvbits represents the similarity between the k-th image patch in the image frame to be processed and the p-th reference image patch in the reference image frame. k，p Lambda represents the number of motion vector bits between the k-th image block in the image frame to be processed and the p-th reference image block in the reference image frame. k This represents the weight factor of the region to be processed corresponding to the kth image block in the image frame to be processed.

[0133] Furthermore, the formula for calculating the weighting factor includes:

[0134] lambda k =exp((qp-delta) k ) / alpha k )

[0135] Where qp represents the preset quantization coefficient, delta k and alpha k These represent the auxiliary weight factors of the region to be processed corresponding to the k-th image block in the image frame to be processed.

[0136] In this optional embodiment, the reference temporal information between the image patch to be processed and each reference image patch in the reference image frame may optionally include the similarity and motion vector between the image patch to be processed and each reference image patch in the reference image frame, cost k，p This represents the cost between the k-th image block in the image frame to be processed and the p-th reference image block in the reference image frame. k，p The mvbits represents the similarity between the k-th image block in the image frame to be processed and the p-th reference image block in the reference image frame. k，p This represents the number of motion vector bits between the k-th image block in the image frame to be processed and the p-th reference image block in the reference image frame.

[0137] In this optional embodiment, the optimal reference image block corresponding to the image block to be processed can be determined based on a local weighting factor strategy. That is, different weighting factors for the region to be processed are determined according to the motion / static type of the region to be processed. The formula for calculating the weighting factor when the motion / static type of the region to be processed is motion type is as follows:

[0138]

[0139] When the dynamic / static type of the region to be processed is static, the formula for calculating the weight factor is:

[0140]

[0141] Lambda k Lambda represents the weight factor of the region to be processed corresponding to the k-th image block in the image frame to be processed. k Including LambdaOri or LambdaOpt, delta k and alpha k Delta and Δ represent the auxiliary weight factors of the region to be processed corresponding to the k-th image block in the image frame to be processed. k Including deltaOri or deltaOpt, alpha k Includes alphaOri or alphaOpt, where qp represents the preset quantization coefficient.

[0142] In this embodiment of the invention, optionally, for example, when the static / dynamic type of the area to be processed includes the motion type, deltaOri = 13.7122, alphaOri = 4.2005; when the static type of the area to be processed includes the motion type, deltaOpt = 13.5, alphaOpt = 4.2. This invention does not impose any limitations.

[0143] As can be seen, implementing this optional embodiment can calculate the reference temporal information between the image block to be processed and each reference image block in the reference image frame. For each image block to be processed, the cost between the image block to be processed and each reference image block in the reference image frame is calculated according to the weight factor of the region to be processed corresponding to the image block to be processed and the reference temporal information, according to the preset cost calculation formula. Based on the strategy of local weight factor, the weight factor can be adjusted according to the dynamic and static state of the region to be processed, and the cost can be calculated based on the weight factor, similarity and number of motion vector bits. This can improve the accuracy and efficiency of determining the best reference image block for the image block to be processed, thereby improving the accuracy and efficiency of subsequent noise reduction of the image block to be processed.

[0144] Example 3

[0145] Please see Figure 3 , Figure 3 This is a schematic diagram of a video image noise reduction device based on temporal information and weighting factors disclosed in an embodiment of the present invention. Figure 3 The described video image denoising device based on temporal information and weighting factors may include an intelligent server or intelligent platform for denoising video images. The intelligent server may include a local server or a cloud server; this embodiment of the invention does not limit the scope. Figure 3 As shown, the video image noise reduction device based on temporal information and weighting factors may include:

[0146] The acquisition module 301 is used to acquire at least one image block to be processed in the image frame to be processed. For each image block to be processed, the best reference image block corresponding to the image block to be processed is determined in the reference image frame corresponding to the image frame to be processed according to a preset weight factor.

[0147] The calculation module 302 is used to calculate the temporal information between the image block to be processed and the corresponding best reference image block for each image block to be processed. The temporal information includes similarity and motion vector.

[0148] The determining module 303 is used to determine the motion intensity of each image block to be processed based on the temporal information corresponding to the image block to be processed, and to determine the reference block filtering coefficient of the best reference image block corresponding to the image block to be processed based on the motion intensity.

[0149] The calculation module 302 is further configured to, for each image block to be processed, calculate the reference pixel filtering coefficient of each reference pixel in the best reference image block according to the reference block filtering coefficient of the best reference image block corresponding to the image block to be processed, and calculate the processing pixel filtering coefficient of each pixel in the image block to be processed according to the reference pixel filtering coefficient of each reference pixel.

[0150] The noise reduction module 304 is used to perform noise reduction on each pixel to be processed in each image block to be processed, based on the pixel to be processed filtering coefficient and the reference pixel filtering coefficient of the corresponding reference pixel, to obtain the noise reduction output result corresponding to the pixel to be processed.

[0151] It is evident that implementation Figure 3 The described video image denoising device based on temporal information and weighting factors can determine the optimal reference image block corresponding to the image block to be processed in the image frame to be processed according to the preset weighting factors. Based on the temporal information between the image block to be processed and the corresponding optimal reference image block, it determines the motion intensity of the image block to be processed. Based on the motion intensity, it determines the reference block filtering coefficient of the optimal reference image block corresponding to the image block to be processed. Based on the reference block filtering coefficient, it calculates the reference pixel filtering coefficient of each reference pixel in the optimal reference image block and the pixel filtering coefficient of each pixel to be processed in the image block to be processed. Based on the pixel filtering coefficient and the reference pixel filtering coefficient, it performs filtering and denoising on the pixel to be processed to obtain the denoising output result corresponding to the pixel to be processed. By calculating temporal information on a block-by-block basis, fewer reference frames can reduce the occupation of additional buffer data, improve the efficiency and accuracy of filtering and denoising for images and videos, and reduce the probability of inter-frame flickering and motion blur problems after video denoising.

[0152] In an optional embodiment, such as Figure 3 As shown, the acquisition module 301 determines the optimal reference image block corresponding to each image block to be processed in the reference image frame corresponding to the image block to be processed according to a preset weight factor in the following specific methods:

[0153] The image frame to be processed and the reference image frame corresponding to the image frame to be processed are divided into regions to obtain at least one region to be processed corresponding to the image frame to be processed and at least one reference region corresponding to the reference image frame. The reference image frame includes at least the previous frame image corresponding to the image frame to be processed, and the region size of the region to be processed is larger than the image block size of the image block to be processed.

[0154] The motion vector histogram of each reference region in the reference image frame is statistically analyzed. For each region to be processed, the motion vector histogram of the reference region corresponding to the region to be processed is used to predict the motion / static type of the region to be processed. The weight factor of the region to be processed is determined based on the motion / static type of the region to be processed. The motion / static type includes motion type or static type.

[0155] For each image block to be processed, the cost between the image block to be processed and each reference image block in the corresponding reference image frame is calculated based on the weight factor of the region to be processed corresponding to the image block to be processed, and the reference image block corresponding to the cost that satisfies the preset selection conditions is determined as the best reference image block corresponding to the image block to be processed.

[0156] It is evident that implementation Figure 3The described video image denoising device based on temporal information and weighting factors can divide the image frame to be processed and the reference image frame into regions, statistically analyze the motion vector histogram of each reference region in the reference image frame, and predict the motion / static type of the region to be processed based on the motion vector histogram, thereby improving the efficiency of determining the motion / static type of the region to be processed. Based on the motion / static type of the region to be processed, a weighting factor is determined for that region. The cost between the image block to be processed and each reference image block in the corresponding reference image frame is calculated based on the weighting factor. The reference image block corresponding to the cost that meets preset selection conditions is determined as the optimal reference image block corresponding to the image block to be processed. This device can determine the optimal reference image block corresponding to the image block to be processed through a local weighting factor strategy, improving the accuracy and efficiency of determining the optimal reference image block while also improving the denoising performance of the image block to be processed. To improve accuracy and efficiency, the system determines the optimal reference image block corresponding to the image block to be processed in the image frame based on a preset weighting factor. Then, based on the temporal information between the image block to be processed and the corresponding optimal reference image block, it determines the motion intensity of the image block to be processed. Based on the motion intensity, it determines the reference block filtering coefficient of the optimal reference image block corresponding to the image block to be processed. Using the reference block filtering coefficient, it calculates the reference pixel filtering coefficient for each reference pixel in the optimal reference image block and the pixel filtering coefficient for each pixel to be processed in the image block to be processed. Finally, it performs filtering and noise reduction on the pixel to be processed based on the pixel filtering coefficient and the reference pixel filtering coefficient, obtaining the noise reduction output result corresponding to the pixel to be processed. This method can improve the efficiency and accuracy of image and video filtering and noise reduction, and reduce the probability of inter-frame flickering and motion blur problems after video noise reduction.

[0157] In another alternative embodiment, such as Figure 3 As shown, the determining module 303 determines the motion intensity of each image block to be processed based on the temporal information corresponding to that image block in the following specific ways:

[0158] Obtain the level thresholds corresponding to the time domain information. The level thresholds include similarity level thresholds and motion vector level thresholds.

[0159] For each image block to be processed, the motion intensity of the image block to be processed is determined based on the target relationship between the temporal information corresponding to the image block to be processed and the level threshold. The target relationship includes the first relationship between the similarity of the image block to be processed and the similarity level threshold and the second relationship between the motion vector of the image block to be processed and the motion vector level threshold.

[0160] The first relationship includes:

[0161]

[0162] The second relationship includes:

[0163]

[0164] The motion intensity of the image patch to be processed includes:

[0165]

[0166] Among them, sadStatus k This represents the first relation corresponding to the k-th image block in the image frame to be processed, sad k mvStatus represents the similarity between the k-th image patch to be processed and the best reference image patch in the image frame to be processed. k This represents the second relation corresponding to the k-th image block in the image frame to be processed, mv k moveStatus represents the motion vector between the k-th image patch to be processed and the best reference image patch in the image frame to be processed. k represents the motion intensity of the k-th image block in the image frame to be processed, sadTh0-sadTh3 represents the similarity level threshold, and mvTh0-mvTh3 represents the motion vector level threshold.

[0167] It is evident that implementation Figure 3 The described video image denoising device based on temporal information and weighting factors can obtain the level thresholds corresponding to the temporal information. The level thresholds include similarity level thresholds and motion vector level thresholds. Based on the target relationship between the temporal information corresponding to the image block to be processed and the level thresholds, the motion intensity of the image block to be processed is determined. The device can determine the motion intensity of the image block to be processed based on the relationship between the similarity and motion vectors corresponding to the image block to be processed and the corresponding threshold levels, thereby improving the accuracy and efficiency of determining the motion intensity of the image block to be processed, and thus improving the denoising efficiency of video images.

[0168] In yet another alternative embodiment, such as Figure 3 As shown, for each image block to be processed, the calculation module 302 calculates the reference pixel filtering coefficients of each reference pixel in the best reference image block according to the reference block filtering coefficients of the best reference image block corresponding to that image block. The specific method includes:

[0169] For each image block to be processed, for each pixel to be processed in the image block to be processed, calculate the absolute value of the difference between the pixel to be processed and each corresponding reference pixel in the best reference image block;

[0170] For each image block to be processed, the extreme value of the absolute value of the difference between the image block to be processed and the corresponding best reference image block is determined in each absolute value of the difference, and the difference between the extreme values ​​of the absolute values ​​of the difference is calculated based on the extreme values ​​of the absolute values ​​of the difference. The extreme values ​​of the absolute values ​​of the difference include the maximum value of the absolute value of the difference and the minimum value of the absolute value of the difference.

[0171] For each image block to be processed, for each pixel to be processed in the image block to be processed, the normalization result of the absolute value of the difference between the pixel to be processed and the corresponding reference pixel within the preset target range is calculated based on the absolute value of the difference between the pixel to be processed and the corresponding reference pixel, the minimum absolute value of the difference, and the extreme value difference of the absolute value of the difference.

[0172] For each image block to be processed, for each pixel to be processed in the image block to be processed, the scaling factor corresponding to the pixel to be processed is calculated based on the normalization result between the pixel to be processed and the corresponding reference pixel. Then, based on the scaling factor and the reference block filtering factor of the best reference image block corresponding to the image block to be processed, the reference pixel filtering factor of each reference pixel in the best reference image block corresponding to the image block to be processed is calculated.

[0173] It is evident that implementation Figure 3 The described video image denoising device based on temporal information and weighting factors can calculate the absolute value of the difference between the pixel to be processed and each corresponding reference pixel in the best reference image block, and determine the extreme values ​​and differences of the absolute values. Based on the absolute values, minimum values, and differences of the absolute values, it calculates the normalized result of the absolute value within a preset target range. Based on the normalized result between the pixel to be processed and the corresponding reference pixel, it calculates the scaling factor corresponding to the pixel to be processed. Based on the scaling factor and the reference block filtering factor of the best reference image block corresponding to the image block to be processed, it calculates the reference pixel filtering factor of each reference pixel in the best reference image block corresponding to the image block to be processed. It can calculate the scaling factor of the exponential model based on the absolute value of the difference of the image block to be processed, and calculate the reference pixel filtering factor of each reference pixel based on the scaling factor, thereby improving the accuracy of the calculated reference pixel filtering factor, thus improving the accuracy and efficiency of filtering and denoising the image block to be processed, and reducing the probability of inter-frame flickering and motion blur problems after video denoising.

[0174] In yet another alternative embodiment, such as Figure 3 As shown, the formula for calculating the normalization result includes:

[0175]

[0176] Among them, normalpixeldiffvalk (i,j) represents the normalized result of the absolute value of the difference between the pixel at coordinates (i,j) in the k-th image block of the image frame and the corresponding reference pixel in the range [-a, a]. k (i,j) represents the absolute value of the difference between the pixel at coordinates (i,j) in the k-th image block and the corresponding reference pixel, minpixeldiff. k The gap represents the minimum absolute value of the difference corresponding to the k-th image patch to be processed. k This represents the absolute value and extreme value difference of the difference corresponding to the k-th image block to be processed;

[0177] Furthermore, the formula for calculating the filter coefficients of each reference pixel in the best reference image block corresponding to each image block to be processed includes:

[0178] pixelRefCoef k (i,j)=refblkCoef k *exp(-normalpixeldiffval k (i,j))

[0179] Among them, pixelRefCoef k (i,j) represents the reference pixel filtering coefficient corresponding to the reference pixel at coordinates (i,j) in the k-th image block to be processed. refblkCoef k The reference block filter coefficients of the best reference image block corresponding to the k-th image block to be processed are exp(-normalpixeldiffval). k (i,j)) represents the scaling factor corresponding to the pixel point with coordinates (i,j) in the k-th image block to be processed;

[0180] Furthermore, the formula for calculating the filter coefficients of each pixel in each image block to be processed includes:

[0181] pixelCurCoef k (i,j)=2 W -pixelRefCoef k (i,j)

[0182] Among them, pixelCurCoef k (i,j) represents the filtering coefficient of the pixel to be processed with coordinates (i,j) in the k-th image block to be processed, and W represents the preset weight value precision.

[0183] It is evident that implementation Figure 3The described video image denoising device based on temporal information and weighting factors can calculate the normalized result of the absolute value of the difference within a preset target range based on the absolute value of the difference, the minimum absolute value of the difference, and the difference between the extreme values ​​of the absolute values ​​of the difference. Then, it calculates the scaling factor of the exponential model based on the normalized result, and calculates the filter coefficient of the reference pixel and the filter coefficient of the pixel to be processed based on the scaling factor. This improves the accuracy and efficiency of the calculation of the pixel filter coefficient, thereby improving the accuracy and efficiency of filtering and denoising the image block to be processed, and reducing the probability of inter-frame flicker and motion blur problems after video denoising.

[0184] In yet another alternative embodiment, such as Figure 3 As shown, for each image block to be processed, the noise reduction module 304 performs noise reduction on each pixel to be processed based on the pixel's filtering coefficients and the filtering coefficients of the corresponding reference pixel, obtaining the noise reduction output result for that pixel in the following specific ways:

[0185] For each image block to be processed, for each pixel to be processed in the image block to be processed, the pixel to be processed is filtered and denoised according to the pixel to be processed filter coefficient, the reference pixel filter coefficient of the reference pixel corresponding to the pixel to be processed and the preset filtering and denoising formula, so as to obtain the denoising output result corresponding to the pixel to be processed.

[0186] The filtering and noise reduction formulas include:

[0187]

[0188] denoiseOut k (i,j)=clip(0,255,denoiseOut k (i,j))

[0189] Among them, denoiseOut k (i,j) represents the noise reduction output of the pixel at coordinates (i,j) in the k-th image block of the image frame to be processed. (curpixelval) k (i,j) represents the pixel value of the pixel at coordinates (i,j) in the k-th image block to be processed, refpixelval k (i,j) represents the pixel value of the reference pixel corresponding to the pixel at coordinates (i,j) in the k-th image block to be processed. clip(0,255,denoiseOut) k(i,j)) means that the noise reduction output of the pixel with coordinates (i,j) in the k-th image block of the image frame to be processed is clamped within [0,255].

[0190] It is evident that implementation Figure 3 The described video image denoising device based on temporal information and weighting factors can filter and denoise the pixel to be processed according to a preset filtering and denoising formula, obtain the denoising output result corresponding to the pixel to be processed, and clamp the denoising output result within [0,255]. The denoising output results of all pixels of the image block to be processed are combined into a new image block to replace the image block to be processed, thus completing the filtering and denoising of the image block to be processed. This can improve the efficiency and accuracy of filtering and denoising images and videos, and reduce the probability of inter-frame flickering and motion blur problems after video denoising.

[0191] In yet another alternative embodiment, such as Figure 3 As shown, the acquisition module 301 calculates the cost between each image block to be processed and each reference image block in the reference image frame based on the weight factor of the region to be processed corresponding to that image block for each image block to be processed, including the following specific methods:

[0192] For each image block to be processed, calculate the reference temporal information between the image block to be processed and each reference image block in the reference image frame;

[0193] For each image block to be processed, the cost between the image block to be processed and each reference image block in the reference image frame is calculated according to the weight factor of the region to be processed corresponding to the image block to be processed and the reference temporal information, according to the preset cost calculation formula.

[0194] The cost calculation formula includes:

[0195] cost k，p =sad k，p +mvbits k，p *Lambda k

[0196] Among them, cost k，p This represents the cost between the k-th image block in the image frame to be processed and the p-th reference image block in the reference image frame. k，p The mvbits represents the similarity between the k-th image block in the image frame to be processed and the p-th reference image block in the reference image frame. k，p Lambda represents the number of motion vector bits between the k-th image block in the image frame to be processed and the p-th reference image block in the reference image frame. kThis represents the weight factor of the region to be processed corresponding to the kth image block in the image frame to be processed.

[0197] Furthermore, the formula for calculating the weighting factor includes:

[0198] lambda k =exp((qp-delta) k ) / alpha k )

[0199] Where qp represents the preset quantization coefficient, delta k and alpha k These represent the auxiliary weight factors of the region to be processed corresponding to the k-th image block in the image frame to be processed.

[0200] It is evident that implementation Figure 3 The described video image denoising device based on temporal information and weighting factors can calculate the reference temporal information between the image block to be processed and each reference image block in the reference image frame. For each image block to be processed, the cost between the image block to be processed and each reference image block in the reference image frame is calculated according to a preset cost calculation formula based on the weighting factor of the region to be processed corresponding to the image block to be processed and the reference temporal information. It can adjust the weighting factor according to the dynamic and static state of the region to be processed based on the local weighting factor strategy, and calculate the cost based on the weighting factor, similarity and motion vector bit count. This can improve the accuracy and efficiency of determining the best reference image block for the image block to be processed, thereby improving the accuracy and efficiency of subsequent denoising of the image block to be processed.

[0201] Example 4

[0202] Please see Figure 4 , Figure 4 This is a schematic diagram of another video image noise reduction device based on temporal information and weighting factors disclosed in an embodiment of the present invention. Figure 4 As shown, the video image noise reduction device based on temporal information and weighting factors may include:

[0203] Memory 401 storing executable program code;

[0204] Processor 402 coupled to memory 401;

[0205] The processor 402 calls the executable program code stored in the memory 401 to execute the steps in the video image denoising method based on temporal information and weighting factors described in Embodiment 1 or Embodiment 2 of the present invention.

[0206] Example 5

[0207] This invention discloses a computer storage medium storing computer instructions. When these computer instructions are invoked, they are used to execute the steps in the video image noise reduction method based on temporal information and weighting factors described in Embodiment 1 or Embodiment 2 of this invention.

[0208] Example 6

[0209] This invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to perform the steps in the video image noise reduction method based on temporal information and weighting factors described in Embodiment 1 or Embodiment 2.

[0210] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0211] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0212] Finally, it should be noted that the video image noise reduction method and apparatus based on temporal information and weighting factors disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video image denoising method based on temporal information and weighting factors, characterized in that, The method includes: At least one image block to be processed is obtained from the image frame to be processed. For each image block to be processed, the best reference image block corresponding to the image block to be processed is determined in the reference image frame corresponding to the image frame to be processed according to a preset weight factor. For each image block to be processed, the temporal information between the image block to be processed and the corresponding best reference image block is calculated, and the temporal information includes similarity and motion vectors; For each image block to be processed, the motion intensity of the image block to be processed is determined according to the temporal information corresponding to the image block to be processed, and the reference block filtering coefficient of the best reference image block corresponding to the image block to be processed is determined according to the motion intensity. For each image block to be processed, the reference pixel filtering coefficient of each reference pixel in the best reference image block is calculated based on the reference block filtering coefficient of the best reference image block corresponding to the image block to be processed, and the filter coefficient of each pixel to be processed in the image block to be processed is calculated based on the reference pixel filtering coefficient of each reference pixel. For each image block to be processed, for each pixel to be processed in the image block to be processed, the pixel to be processed is filtered and denoised according to the pixel to be processed filter coefficient and the reference pixel filter coefficient of the corresponding reference pixel to obtain the denoised output result corresponding to the pixel to be processed.

2. The video image denoising method based on temporal information and weighting factors according to claim 1, characterized in that, For each image block to be processed, determining the optimal reference image block corresponding to the image block to be processed in the reference image frame corresponding to the image frame to be processed according to a preset weighting factor includes: The image frame to be processed and the reference image frame corresponding to the image frame to be processed are divided into regions to obtain at least one region to be processed corresponding to the image frame to be processed and at least one reference region corresponding to the reference image frame. The reference image frame includes at least the previous frame image corresponding to the image frame to be processed. The region size of the region to be processed is larger than the image block size of the image block to be processed. The motion vector histogram of each reference region in the reference image frame is statistically analyzed. For each region to be processed, the motion vector histogram of the reference region corresponding to the region to be processed is used to predict the motion / static type of the region to be processed. The weight factor of the region to be processed is determined based on the motion / static type of the region to be processed. The motion / static type includes motion type or static type. For each image block to be processed, the cost between the image block to be processed and each reference image block in the corresponding reference image frame is calculated based on the weight factor of the region to be processed corresponding to the image block to be processed, and the reference image block corresponding to the cost that satisfies the preset selection conditions is determined as the best reference image block corresponding to the image block to be processed.

3. The video image denoising method based on temporal information and weighting factors according to claim 2, characterized in that, For each image block to be processed, determining the motion intensity of the image block according to its corresponding temporal information includes: Obtain the level threshold corresponding to the time domain information, wherein the level threshold includes a similarity level threshold and a motion vector level threshold; For each image block to be processed, the motion intensity of the image block to be processed is determined according to the target relationship between the temporal information corresponding to the image block to be processed and the level threshold. The target relationship includes a first relationship between the similarity of the image block to be processed and the similarity level threshold and a second relationship between the motion vector of the image block to be processed and the motion vector level threshold. The first relationship includes: The second relationship includes: The motion intensity of the image patch to be processed includes: Among them, sadStatus k This represents the first relationship corresponding to the k-th image block in the image frame to be processed, sad k mvStatus represents the similarity between the k-th image patch to be processed in the image frame to be processed and the best reference image patch. k The second relation, mv, represents the k-th image block in the image frame to be processed. k moveStatus represents the motion vector between the k-th image block to be processed and the best reference image block in the image frame to be processed. k The motion intensity of the k-th image block in the image frame to be processed is represented by sadTh0-sadTh3, the similarity level threshold is represented by sadTh0-sadTh3, and the motion vector level threshold is represented by mvTh0-mvTh3.

4. The video image denoising method based on temporal information and weighting factors according to any one of claims 1-3, characterized in that, For each of the image blocks to be processed, the reference pixel filtering coefficients of each reference pixel in the best reference image block are calculated based on the reference block filtering coefficients of the best reference image block corresponding to that image block, including: For each image block to be processed, for each pixel to be processed in the image block to be processed, calculate the absolute value of the difference between the pixel to be processed and each corresponding reference pixel in the best reference image block; For each image block to be processed, the extreme value of the absolute value of the difference between the image block to be processed and the corresponding best reference image block is determined in each absolute value of the difference, and the extreme value difference of the absolute value of the difference is calculated based on the extreme value of the absolute value of the difference, wherein the extreme value of the absolute value of the difference includes the maximum value of the absolute value of the difference and the minimum value of the absolute value of the difference; For each image block to be processed, for each pixel to be processed in the image block to be processed, the normalization result of the absolute difference between the pixel to be processed and the corresponding reference pixel within a preset target range is calculated based on the absolute value of the difference between the pixel to be processed and the corresponding reference pixel, the minimum value of the absolute difference, and the extreme value difference of the absolute difference. For each image block to be processed, for each pixel to be processed in the image block to be processed, the scaling factor corresponding to the pixel to be processed is calculated based on the normalization result between the pixel to be processed and the corresponding reference pixel. Then, based on the scaling factor and the reference block filtering factor of the best reference image block corresponding to the image block to be processed, the reference pixel filtering factor of each reference pixel in the best reference image block corresponding to the image block to be processed is calculated.

5. The video image denoising method based on temporal information and weighting factors according to claim 4, characterized in that, The formula for calculating the normalization result includes: Among them, normalpixeldiffval k (i,j) represents the normalized result of the absolute value of the difference between the pixel at coordinates (i,j) in the k-th image block of the image frame to be processed and the corresponding reference pixel in the range [-a, a]. k (i,j) represents the absolute value of the difference between the pixel at coordinates (i,j) in the k-th image block to be processed and the corresponding reference pixel, minpixeldiff k The gap represents the minimum absolute value of the difference corresponding to the k-th image block to be processed. k This represents the absolute value and extreme value difference of the difference corresponding to the k-th image block to be processed; Furthermore, the formula for calculating the filter coefficients of each reference pixel in the optimal reference image block corresponding to each image block to be processed includes: pixelRefCoef k (i,j)=refblkCoef k *exp(-normalpixeldiffval k (i,j)) Among them, pixelRefCoef k (i,j) represents the reference pixel filtering coefficient corresponding to the reference pixel at coordinates (i,j) in the k-th image block to be processed. refblkCoef k The reference block filter coefficients of the optimal reference image block corresponding to the k-th image block to be processed are exp(-normalpixeldiffval). k (i,j)) represents the scaling factor corresponding to the pixel point with coordinates (i,j) in the k-th image block to be processed; Furthermore, the formula for calculating the filter coefficients of each pixel in each image block to be processed includes: pixelCurCoef k (i,j)=2 W -pixelRefCoef k (i,j) Among them, pixelCurCoef k (i,j) represents the filtering coefficient of the pixel to be processed at coordinates (i,j) in the k-th image block to be processed, and W represents the preset weight value precision.

6. The video image denoising method based on temporal information and weighting factors according to claim 5, characterized in that, For each image block to be processed, for each pixel to be processed within that image block, filtering and noise reduction are performed on the pixel to be processed based on its filtering coefficients and the filtering coefficients of the reference pixel corresponding to it, to obtain the noise reduction output result corresponding to the pixel to be processed, including: For each image block to be processed, for each pixel to be processed in the image block to be processed, the pixel to be processed is filtered and denoised according to the pixel to be processed filter coefficient, the reference pixel filter coefficient of the reference pixel corresponding to the pixel to be processed and the preset filtering and denoising formula, so as to obtain the denoising output result corresponding to the pixel to be processed. The filtering and noise reduction formula includes: denoiseOut k (i,j)=clip(0,255,denoiseOut k (i,j)) Among them, denoiseOut k (i,j) represents the noise reduction output of the pixel at coordinates (i,j) in the k-th image block of the image frame to be processed, curpixelval k (i,j) represents the pixel value of the pixel at coordinates (i,j) in the k-th image block to be processed, refpixelval k (i,j) represents the pixel value of the reference pixel corresponding to the pixel at coordinates (i,j) in the k-th image block to be processed. clip(0,255,denoiseOut) k (i,j)) indicates that the noise reduction output of the pixel with coordinates (i,j) in the k-th image block of the image frame to be processed is clamped within [0,255].

7. The video image denoising method based on temporal information and weighting factors according to claim 2 or 3, characterized in that, For each image block to be processed, the cost between the image block to be processed and each reference image block in the reference image frame is calculated based on the weight factor of the region to be processed corresponding to the image block to be processed, including: For each image block to be processed, calculate the reference temporal information between the image block to be processed and each reference image block in the reference image frame; For each image block to be processed, the cost between the image block to be processed and each reference image block in the reference image frame is calculated according to the weight factor of the region to be processed corresponding to the image block to be processed and the reference temporal information, based on a preset cost calculation formula. The cost calculation formula includes: cost k，p =sad k，p +mvbits k，p *Lambda k Among them, cost k，p This represents the cost between the k-th image block to be processed in the image frame to be processed and the p-th reference image block in the reference image frame, sad k，p The mvbits represents the similarity between the k-th image block in the image frame to be processed and the p-th reference image block in the reference image frame. k，p Lambda represents the number of motion vector bits between the k-th image block in the image frame to be processed and the p-th reference image block in the reference image frame. k The weight factor represents the region to be processed corresponding to the k-th image block in the image frame to be processed; Furthermore, the formula for calculating the weighting factor includes: Lambda k =exp((qp-delta k ) / alpha k ) Where qp represents the preset quantization coefficient, delta k and alpha k These represent the auxiliary weight factors of the region to be processed corresponding to the k-th image block in the image frame to be processed.

8. A video image noise reduction device based on temporal information and weighting factors, characterized in that, The device includes: The acquisition module is used to acquire at least one image block to be processed in the image frame to be processed. For each image block to be processed, the best reference image block corresponding to the image block to be processed is determined in the reference image frame corresponding to the image frame to be processed according to a preset weight factor. The calculation module is used to calculate, for each image block to be processed, the temporal information between the image block to be processed and the corresponding best reference image block, wherein the temporal information includes similarity and motion vector; The determination module is used to determine the motion intensity of each image block to be processed based on the temporal information corresponding to the image block to be processed, and to determine the reference block filtering coefficient of the best reference image block corresponding to the image block to be processed based on the motion intensity. The calculation module is further configured to, for each image block to be processed, calculate the reference pixel filtering coefficient of each reference pixel in the best reference image block according to the reference block filtering coefficient of the best reference image block corresponding to the image block to be processed, and calculate the pixel filtering coefficient of each pixel to be processed in the image block to be processed according to the reference pixel filtering coefficient of each reference pixel. The noise reduction module is used to perform noise reduction on each pixel to be processed in each image block to be processed, based on the pixel to be processed filtering coefficient and the reference pixel filtering coefficient of the reference pixel corresponding to the pixel to be processed, to obtain the noise reduction output result corresponding to the pixel to be processed.

9. A video image noise reduction device based on temporal information and weighting factors, characterized in that, The device includes: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the video image noise reduction method based on temporal information and weighting factors as described in any one of claims 1-7.

10. A computer storage medium, characterized in that, The computer storage medium stores computer instructions, which, when invoked, are used to execute the video image noise reduction method based on temporal information and weighting factors as described in any one of claims 1-7.

Images