Video frame image denoising method, device, storage medium and program product

Abstract

The present invention discloses a video frame image denoising method, device, storage medium and program product, which relates to the field of video image processing technology. The method comprises: performing spatial domain denoising on the current frame image in color space to obtain an intermediate image; performing motion compensation on the denoised image of the previous frame based on the intermediate image to obtain a compensated frame image; determining a compensation error based on the compensated frame image and the current frame image; determining the estimated noise of the current frame image based on the intermediate image and the current frame image; determining the filter coefficient of the time domain filter based on the estimated noise and the compensation error; performing time domain denoising on the current frame image based on the time domain filter to obtain a denoised image corresponding to the current frame image. Using the scheme provided by the present application, the video frame image is first subjected to spatial domain denoising and then subjected to time domain denoising, and the denoising effect is effectively improved by taking into account the influence of the random characteristics of inter-frame noise.

Description

Technical Field

[0001] The present invention relates to the technical field of video image processing, and in particular to a video frame image denoising method, device, storage medium and program product. Background Art

[0002] Nowadays, optical endoscopes play a key role in the field of medical diagnosis. However, the small size of optical endoscopes limits the size of photoelectric imaging sensors, resulting in large imaging noise, which seriously affects the signal-to-noise ratio of the image and may be difficult to meet clinical needs.

[0003] In the related art, for the scanned video images, continuous single-frame images are usually filtered in the intra-frame spatial domain, for example, the single-frame video images are filtered by bilateral filtering, wavelet denoising, etc., so as to improve the image quality.

[0004] However, in the solutions provided by the related art, frame filtering is only performed on a single frame image, and the random characteristics of noise between video frames are not taken into account, resulting in poor denoising effect and poor quality of generated video frame images. Summary of the invention

[0005] The purpose of the present invention is to provide a video frame image denoising method, device, storage medium and program product, and the purpose is to combine spatial domain denoising with temporal domain denoising in the process of video frame image denoising to improve the denoising effect of video frame image, which is beneficial to improving the image quality of the generated image.

[0006] To achieve the above objectives, on the one hand, the present application provides a video frame image denoising method, the method comprising:

[0007] Perform spatial domain denoising on the current frame image in the color space to obtain an intermediate image;

[0008] Performing motion compensation on the denoised image of the previous frame based on the intermediate image to obtain a compensated frame image;

[0009] Determine a compensation error based on the compensation frame image and the current frame image;

[0010] Determining estimated noise of the current frame image based on the intermediate image and the current frame image;

[0011] determining filter coefficients of a time domain filter based on the estimated noise and the compensated error;

[0012] The current frame image is subjected to time domain denoising based on the time domain filter to obtain a denoised image corresponding to the current frame image.

[0013] As a further improvement of the present application, determining the filter coefficient of the time domain filter based on the estimated noise and the compensation error includes:

[0014] determining an interpolation factor based on the compensation error and the estimated noise;

[0015] A first filter coefficient and a second filter coefficient of the time domain filter are determined based on the interpolation factor and an adjustment parameter, wherein the adjustment parameter is used to control a balance between error compensation and noise suppression of the time domain filter.

[0016] As a further improvement of the present application, determining the interpolation factor based on the compensation error and the estimated noise includes:

[0017] Determine the ratio of the compensation error to the estimated noise as the interpolation factor;

[0018] The determining the first filter coefficient and the second filter coefficient of the time domain filter based on the interpolation factor and the adjustment parameter includes:

[0019] Determine the first filter coefficient based on a curve formula formed by performing a power operation on the interpolation factor and the adjustment parameter;

[0020] The second filter coefficient is determined based on the first filter coefficient, wherein a sum of the first filter coefficient and the second filter coefficient is 1, or a sum of the first filter coefficient and the second filter coefficient is not 1.

[0021] As a further improvement of the present application, in the step of “determining the first filter coefficient based on a curve formula formed by performing a power operation on the interpolation factor and the adjustment parameter”, the formula for determining the first filter coefficient is:

[0022] or,

[0023] In the step of “determining the second filter coefficient based on the first filter coefficient”, the formula for determining the second filter coefficient is:

[0024] coef2=1-coef1, or coef2=scale*(1-coef1)

[0025] Among them, coef1 represents the first filter coefficient, coef2 represents the second filter coefficient, curve_th represents the adjustment parameter, and scale represents the configurable hyperparameter.

[0026] As a further improvement of the present application, the step of performing motion compensation on the denoised image of the previous frame based on the intermediate image to obtain a compensated frame image includes:

[0027] Traversing each target pixel in the intermediate image in a manner of row interval w and column interval w, performing a similarity search on the denoised image based on the target pixel, and determining a best matching point of the target pixel, wherein the best matching point has the highest similarity with the best matching point;

[0028] The pixel data in the first ROI window in the denoised image is copied to the second ROI window in the compensated frame image, wherein the first ROI window is centered on the best matching point, and the second ROI window is centered on the compensated pixel point at the coordinates of the target pixel point, and both the first ROI window and the second ROI window contain w×w pixels.

[0029] As a further improvement of the present application, the performing similarity search on the denoised image based on the target pixel point to determine the best matching point of the target pixel point includes:

[0030] In the denoised image, taking the first candidate pixel point at the target pixel point coordinate as the starting point, performing N path searches;

[0031] After the Nth path search is completed, the similarity between each candidate pixel point in the search path and the target pixel point is calculated;

[0032] The candidate pixel point in the search path that has the highest similarity to the target pixel point is determined as the best matching point.

[0033] As a further improvement of the present application, in the denoised image, taking the first candidate pixel point at the target pixel point coordinate as the starting point, performing N path searches, including:

[0034] In the case of the first path search, taking the first candidate pixel point as the center, searching for a second candidate pixel point on a first radius;

[0035] Calculating the similarity between each second candidate pixel point in the search path and the target pixel point;

[0036] In the case of performing the i-th path search, taking the i-th candidate point having the highest similarity to the target pixel point as the center, searching for the i+1-th candidate pixel point on the i-th radius;

[0037] Calculate the similarity between each i+1th candidate pixel point in the search path and the target pixel point;

[0038] The i-th radius of the i-th path search is greater than the i+1-th radius of the i+1-th path search.

[0039] As a further improvement of the present application, the calculating the similarity between each candidate pixel point in the search path and the target pixel point includes:

[0040] Calculate the absolute value of the difference between the Y channel component of each pixel in the third ROI window in the denoised image and the fourth ROI window in the intermediate image, wherein the third ROI window is centered on the candidate pixel point, and the fourth ROI window is centered on the target pixel point;

[0041] The sum of the absolute values ​​of the differences of the Y channel components of each pixel is determined as the similarity.

[0042] As a further improvement of the present application, the determining the compensation error based on the compensation frame image and the current frame image includes:

[0043] Traversing the target pixel points in the current frame image and the compensation frame pixel points in the compensation frame image;

[0044] Calculate the sum of the absolute values ​​of the differences between the Y channel components of each pixel point in the fifth ROI window in the current frame image and the sixth ROI window in the compensation frame image, wherein the fifth ROI window is centered on the target pixel point and the sixth ROI window is centered on the compensation frame pixel point;

[0045] Calculate the mean value of the Y channel component of each pixel point in the fifth ROI window;

[0046] The ratio of the sum of the absolute values ​​of the differences of the Y channel components of each pixel point to the mean value is determined as the compensation error corresponding to the target pixel point.

[0047] As a further improvement of the present application, the traversing the target pixel points in the current frame image and the compensation frame pixel points in the compensation frame image includes:

[0048] Traversing the target pixel points in the current frame image and the compensated frame pixel points in the compensated image in a row interval of a;

[0049] The method further comprises:

[0050] For the target pixel point that has not been traversed, linear interpolation is performed on the compensation errors corresponding to two adjacent target pixel points in the pixel row to determine the compensation error corresponding to the target pixel point that has not been traversed.

[0051] As a further improvement of the present application, determining the estimated noise of the current frame image based on the intermediate image and the current frame image includes:

[0052] Calculate the absolute value of the difference between the Y channel components of each pixel in the current frame image and the intermediate image, and determine it as the pixel difference;

[0053] Each pixel point of the current frame image is traversed, and the variance of the pixel difference value corresponding to each pixel in different seventh ROI windows is calculated to obtain the estimated noise corresponding to each pixel point, and the seventh ROI window is centered on the traversed pixel point.

[0054] As a further improvement of the present application, performing spatial domain denoising on the current frame image in the color space to obtain an intermediate image includes:

[0055] Convert the current frame image from the RGB color space to the YUV color space to obtain a Y channel component, a U channel component, and a V channel component;

[0056] The Y channel component is subjected to denoising processing, and the U channel component and the V channel component are maintained to obtain the intermediate image.

[0057] As a further improvement of the present application, performing time domain denoising on the current frame image based on the time domain filter to obtain a denoised image corresponding to the current frame image includes:

[0058] Based on the time domain filter, the Y channel component, the U channel component and the V channel component of the current frame image are filtered respectively to obtain the filtered denoised image.

[0059] On the other hand, the present application provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps in the video frame image denoising method as described in any of the above aspects.

[0060] On the other hand, the present application provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the steps in the video frame image denoising method as described in any of the above aspects. On the other hand, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps in the video frame image denoising method as described in any of the above aspects. The solution provided in the embodiment of the present application has at least the following features:

[0061] Beneficial effects:

[0062] In an embodiment of the present application, the current image is first subjected to spatial denoising, and after motion compensation is performed based on the obtained intermediate image and the denoised image of the previous frame, the compensation error is determined, and the filter coefficient of the time domain filter is determined based on the compensation error and the estimated noise of the current frame image, and finally the current frame image is filtered based on the time domain filter to obtain a denoised image. In the case where the noise between video frames has random characteristics, spatial denoising is combined with time domain denoising to effectively suppress the noise of the video frame image. In addition, in the present application, the filter coefficient is determined based on the compensation error and the estimated noise, and the filter coefficient of the time domain filter can be dynamically adjusted, thereby effectively avoiding motion artifacts caused by poor matching due to rapid motion. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 A flowchart of a video frame image denoising method provided by an exemplary embodiment of the present application is shown;

[0064] Figure 2 A schematic diagram showing the relationship between a first filter coefficient and an interpolation factor provided schematically in the present application is shown;

[0065] Figure 3 A flowchart of a process for determining a compensation error provided by an exemplary embodiment of the present application is shown;

[0066] Figure 4 A schematic diagram of a three-step path search corresponding to a pixel point provided by an exemplary embodiment of the present application is shown;

[0067] Figure 5 A flowchart of a noise estimation calculation process provided by an exemplary embodiment of the present application is shown;

[0068] Figure 6 A flowchart of a video frame image denoising process provided by an exemplary embodiment of the present application is shown;

[0069] Figure 7 A structural block diagram of a computer device provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0070] The present invention will be described in detail below in conjunction with the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional changes made by a person skilled in the art based on these embodiments are all within the scope of protection of the present invention.

[0071] It should be noted that the term "comprises" or any other variation thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In addition, the terms "first", "second", etc. are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0072] It should be noted that the solution provided in the embodiment of the present application is executed by an electronic device, which may be a personal computer, a laptop, a server, a digital video recorder, a smart mobile device, etc. In the following embodiment, a computer device is taken as an example to illustrate the denoising method of a video frame image. Figure 1 , which shows a flow chart of a video frame image denoising method provided by an exemplary embodiment of the present application, the method comprising the following steps:

[0073] Step 101, perform spatial domain denoising on the current frame image in color space to obtain an intermediate image.

[0074] Color space is used to describe and identify the range and characteristics of colors, specifying colors through a set of numbers or parameters.

[0075] In a possible implementation, spatial domain denoising is performed on the current frame image in the YUV color space. First, the current frame image is converted from the RGB color space to the YUV color space to obtain a Y channel component, a U channel component, and a V channel component. Subsequently, the Y channel component is denoised, and the U channel component and the V channel component are retained to obtain an intermediate image.

[0076] Among them, the Y channel represents brightness information, and the U and V channels represent chrominance information respectively. Spatial denoising can be performed by means of mean filtering, bilateral filtering, median filtering, anisotropic diffusion, local Laplace filtering or Gaussian filtering. The specific denoising method can be selected according to the characteristics of the image noise to ensure the quality of the denoised image. During processing, only the Y channel component is spatially filtered, and the U and V channels remain unchanged, which can avoid the loss of chrominance information, and the obtained image is the intermediate image. Spatial filtering only on the Y channel component can effectively reduce the computational complexity of spatial filtering, and subsequent processing can also be performed on the Y channel component to simplify the computational complexity.

[0077] Optionally, this step may be performed on the current frame image in other similar color spaces, such as HSV, HSL, and LAB color spaces.

[0078] Step 102: Perform motion compensation on the denoised image of the previous frame based on the intermediate image to obtain a compensated frame image.

[0079] Since there are usually similar contents between adjacent frames in a video frame sequence, the movement of the same object in different video frames causes the displacement of pixels between adjacent frames. The purpose of motion compensation is to estimate and compensate for this pixel displacement.

[0080] In one possible implementation, the computer device, in the process of performing motion compensation on the denoised image of the previous frame, first selects a target pixel in the intermediate image, and performs a similarity search on the denoised image of the previous frame based on the target pixel to determine the pixel area in the previous frame that is most similar to the area where the target pixel in the current frame is located.

[0081] Step 103: determining a compensation error based on the compensation frame image and the current frame image.

[0082] Optionally, the computer device traverses the pixel points in the current image and the compensation frame image, calculates the pixel difference between the corresponding positions of the two, and thus determines the compensation error.

[0083] Furthermore, in order to improve the calculation accuracy of the compensation error, the computer device uses a ROI (Region Of Interest) window to limit the calculation area.

[0084] The compensation error is used to evaluate the effect of motion compensation, which is helpful to further optimize the denoising effect.

[0085] Step 104: Determine the estimated noise of the current frame image based on the intermediate image and the current frame image.

[0086] The intermediate image is the result of spatial denoising of the current frame image in color space, while the current frame image is the image that has not been denoised. Therefore, the pixel difference between the corresponding positions of the two can represent the estimated noise of the current frame image.

[0087] Optionally, the computer device selects an appropriate ROI window to limit the calculation area, thereby improving the accuracy of noise estimation. Since the accuracy of noise estimation is crucial to the denoising effect, it is necessary to select an appropriate ROI window size and calculation method.

[0088] Step 105 : determining filter coefficients of a time domain filter based on the estimated noise and the compensated error.

[0089] The purpose of filtering the image is to reduce the influence of estimation noise and compensation error on the final imaging as much as possible. Optionally, the least mean square algorithm, Kalman filtering method and impulse response filtering method can be used as the filtering algorithm of the time domain filter.

[0090] Optionally, the filter type may be determined first, and then corresponding filter coefficients may be calculated according to the selected filter type.

[0091] Step 106: perform time domain denoising on the current frame image based on a time domain filter to obtain a denoised image corresponding to the current frame image.

[0092] After obtaining the filter coefficient, the computer device uses the time domain filter corresponding to the time domain filter coefficient to perform denoising on the current frame image to reduce image quality problems caused by noise.

[0093] Optionally, a time domain filter is usually applied between consecutive frames of an image sequence to further reduce noise in the time domain by performing weighted averaging or filtering on the image, thereby obtaining a denoised image and improving the quality of the final video frame image.

[0094] In summary, in the embodiment of the present application, the current image is first subjected to spatial denoising, and after motion compensation is performed based on the obtained intermediate image and the denoised image of the previous frame, the compensation error is determined, and the filter coefficient of the time domain filter is determined based on the compensation error and the estimated noise of the current frame image, and finally the current frame image is filtered based on the time domain filter to obtain a denoised image. In the case where the noise between video frames has random characteristics, spatial denoising is combined with time domain denoising to effectively suppress the noise of the video frame image. In addition, in the present application, the filter coefficient is determined based on the compensation error and the estimated noise, and the filter coefficient of the time domain filter can be dynamically adjusted, thereby effectively avoiding motion artifacts caused by poor matching due to rapid motion.

[0095] In the embodiment of the present application, in order to better filter out the temporal noise of the current frame image, the computer device needs to be able to accurately determine the filter coefficient of the temporal filter used. The calculation method for determining the temporal filter coefficient is described below through an illustrative embodiment.

[0096] First, the interpolation factor is determined based on the compensation error and the estimated noise.

[0097] The interpolation factor is obtained by quantifying the relationship between the compensation error and the estimated noise by the computer device, and the size of the interpolation factor reflects the relative influence of the compensation error and the noise in the image.

[0098] The compensation error reflects the inaccuracy in the motion compensation process, while the estimated noise represents the noise introduced into the image due to factors such as the sensor and the environment. The interpolation factor quantifies the degree of influence of the noise and compensation error in the image on the image quality. The larger the interpolation factor, the larger the compensation error, and the image requires more compensation processing; the smaller the interpolation factor, the larger the noise, and the more urgent the need for denoising. Calculating the filter coefficient based on the difference factor is conducive to the adaptive adjustment of the time domain filter performance of the computer equipment.

[0099] Optionally, the ratio of the compensation error to the estimated noise is determined as the interpolation factor. The calculation formula of the difference factor is as follows:

[0100]

[0101] Among them, I represents the interpolation factor, error_comp represents the compensation error, and noise represents the estimated noise.

[0102] From the above formula, we can see that when the compensation error is large, the interpolation factor is also large, indicating that the effect of motion compensation is poor, and when the noise is large, the interpolation factor is small, indicating that stronger noise suppression is needed. The choice of this interpolation factor directly affects the calculation result of the filter coefficient in the time domain filter, thereby affecting the denoising effect.

[0103] After the difference factor is obtained, a first filter coefficient and a second filter coefficient of the time domain filter are determined based on the interpolation factor and the adjustment parameter.

[0104] By combining the interpolation factor with the adjustment parameter, the filter can dynamically adjust the weights of these two coefficients according to the relative size of the compensation error and noise.

[0105] Among them, the adjustment parameter is used to control the balance between the compensation error and the noise suppression of the time domain filter, that is, according to the size of the difference factor, the influence of noise suppression and compensation error is balanced in the filter coefficient. When the compensation error is large, the first filter coefficient can be made relatively large by adjusting the parameters to strengthen the correction of the compensation error; and when the noise is large, the second filter coefficient can be made relatively large to enhance the effect of noise suppression.

[0106] The first filter coefficient and the second filter coefficient are two main parameters of the time domain filter, which are used to control the response of the filter to noise and compensation error when processing the current frame image.

[0107] Optionally, the computer device determines the first filter coefficient based on a curve formula formed by performing a power operation on the interpolation factor and the adjustment parameter. Subsequently, the second filter coefficient is determined based on the first filter coefficient. That is, the size of the filter coefficient is determined by the power relationship between the interpolation factor and the adjustment parameter.

[0108] The combination of the interpolation factor and the adjustment parameter adjusts the sensitivity of the filter to the compensation error through the power operation. When the interpolation factor is large, it means that the compensation error is large. At this time, the result of the power operation will enhance the first filter coefficient (weight factor), so that the time domain filter can enhance the correction ability of the compensation error. The role of the adjustment parameter is to further refine this relationship and control the balance of the compensation error correction to avoid excessive suppression of image details.

[0109] For illustration, please refer to Figure 2 , which shows a schematic diagram of the relationship between the first filter coefficient and the interpolation factor provided by the present application. The horizontal axis represents the interpolation factor I, and the vertical axis represents the first filter coefficient (weight factor coef1). The weight factor is positively correlated with the interpolation factor, that is, the larger the interpolation factor, the larger the corresponding first filter coefficient.

[0110] Optionally, the calculation formula of the first filter coefficient is:

[0111] or,

[0112] Among them, coef1 represents the first filter coefficient, curve_th represents the adjustment parameter, and scale represents the configurable hyperparameter.

[0113] When the first filter coefficient is obtained, the second filter coefficient is determined based on the first filter coefficient.

[0114] Optionally, the sum of the first filter coefficient and the second filter coefficient is 1, or the sum of the first filter coefficient and the second filter coefficient is not 1. Whether the sum of the first filter coefficient and the second filter coefficient is 1 depends on the required denoising effect and the relative influence of noise and compensation error in the image.

[0115] The calculation formula of the second coefficient is as follows:

[0116] coef2=1-coef1, or coef2=scale*(1-coef1)

[0117] Among them, scale represents a configurable hyperparameter, and coef2 represents the second filter coefficient.

[0118] When the filter needs stronger noise suppression, the second filter coefficient may occupy a larger proportion, and the first filter coefficient is smaller; on the contrary, if the compensation error is large, the first filter coefficient may be larger. By adjusting the sum of these two coefficients, the filter can maintain the structure and details of the image while removing noise, avoiding excessive smoothing or distortion.

[0119] In a possible implementation, after determining the time domain filter coefficient, the Y channel component, the U channel component and the V channel component of the current frame image are filtered based on the time domain filter to obtain a filtered denoised image.

[0120] First, the current frame image is converted from the RGB color space to the YUV color space to obtain the Y channel component, the U channel component, and the V channel component.

[0121] The temporal filter is a tool for removing noise by filtering pixels between adjacent frames in a time or image sequence. In this embodiment, the temporal filter processes not only the brightness information (Y channel) of the image, but also the chrominance information (U and V channels) to ensure the balance and consistency of the color and brightness of the image after the denoising process.

[0122] Subsequently, when performing time domain filtering, the time domain filter determines the filter coefficients based on the noise estimation and compensation error of the Y channel component of the current frame image, and performs denoising on the Y channel. The time domain filter can use weighted averaging, low-pass filtering and other techniques to remove noise in the Y channel while maintaining the details of the image.

[0123] Similarly, the U and V channel components are denoised through similar processing. Although the U and V channels have less impact on the image than the Y channel, they are still crucial to the color fidelity, so proper denoising of these two channels can ensure the accuracy and stability of the image color information. The denoising of the U and V channels can use a similar filtering method to the Y channel to ensure that important color information is not lost while removing noise.

[0124] In the embodiment of the present application, by combining the difference factor with the adjustment parameter, a method for determining the time domain filter coefficient is provided, which can dynamically adjust the filter coefficient according to the noise estimation of the current frame image and the motion compensation error, and achieve the best denoising effect for different current frame images, thereby significantly improving the quality of the video image. This method can be widely used in the fields of video processing, image restoration, etc., especially in the field of dynamic video denoising, and has significant advantages.

[0125] Before determining the time domain filter coefficient, the compensation error and the estimated noise need to be determined first, so as to design the filter coefficient based on the compensation error and the estimated noise. The following is an explanation of the compensation error determination process and the estimated noise calculation process respectively through exemplary embodiments.

[0126] 1. Compensation Error

[0127] Please refer to Figure 3 , which shows a flowchart of a process for determining a compensation error provided by an exemplary embodiment of the present application, the process comprising the following steps:

[0128] Step 301, traverse each target pixel in the intermediate image with a row interval of w and a column interval of w, perform a similarity search in the denoised image based on the target pixel, and determine the best matching point of the target pixel.

[0129] Among them, the best matching point has the highest similarity with the target pixel point, and the target pixel point refers to each pixel point in the intermediate image. The row interval w and column interval w refer to the spacing between adjacent pixels when performing motion compensation, and the setting of this interval can be optimized according to the resolution and computing power of the image. Traverse each target pixel point to ensure that every part of the image can be compensated.

[0130] Optionally, the purpose of the similarity search performed by the computer device is to find the pixel point that is most similar to the current target pixel point in the denoised image, and to determine the corresponding area in the previous frame image, so as to map and compensate the pixels during the motion compensation process.

[0131] Optionally, the best matching pixel is selected based on the characteristics of the target pixel and the characteristics of the pixel in the denoised image. The best matching point has the highest similarity with the target pixel, thereby ensuring the accuracy of compensation.

[0132] In a possible implementation, multiple path searches are required to determine the best matching point. The process of determining the best matching point is as follows:

[0133] Step A: In the denoised image, taking the first candidate pixel point at the target pixel point coordinate as the starting point, perform N path searches.

[0134] Path search starts from a candidate pixel in the denoised image and gradually expands the search range to find the area most similar to the current target pixel. In each path search, the search radius gradually decreases and the overall search area gradually increases to ensure that the best matching point can be found.

[0135] The first candidate pixel is used as the starting point, usually with the surrounding neighborhood of the target pixel as a reference, to start the search process by preliminarily estimating possible matching positions.

[0136] Optionally, in the case of the first path search, the second candidate pixel point within the first radius is searched with the first candidate pixel point as the center. The path search gradually expands the search range to find the pixel point most similar to the target pixel point. In the first path search, the second candidate pixel point within the first radius is determined with the first candidate pixel point as the center. By calculating the similarity between these candidate pixels and the target pixel point, it is determined which pixels may be the matching points closest to the target pixel point. After the second candidate pixel point is determined, the similarity between these pixels and the target pixel point is calculated, usually evaluated by the difference in pixel values ​​or the structural similarity measurement method. Pixels with higher similarity are considered to be matching points that are closer to the target pixel point.

[0137] Similarly, when performing the i-th path search, the i+1 candidate pixel point on the i-th radius is searched with the i-th candidate point with the highest similarity to the target pixel point as the center. During the i-th path search, the candidate point with the highest similarity is selected as the new search center, and the search radius is expanded on this basis to find the i+1 candidate pixel point. In this process, each expansion of the search range will gradually and accurately find the best matching point based on the previous matching results. After searching for the i+1 pixel point, the similarity between each i+1 candidate pixel point in the search path and the target pixel point is calculated. As the number of path searches increases, the overall search range and accuracy are gradually improved, and ultimately the target pixel point and the best matching point in the compensated image are accurately matched.

[0138] Among them, the i-th radius of the i-th path search is greater than the i+1-th radius of the i+1-th path search. That is, the search range is gradually narrowed, starting from a larger radius and gradually focusing on a more precise area. Through this strategy, the initial search can cover a wider range of candidate areas, and as the search deepens, the scope of a single search is narrowed to improve the search accuracy. This is conducive to ensuring that possible matching points are found in a larger area, and ultimately accurately determining the best matching point in a smaller area, thereby improving the effect of motion compensation.

[0139] It should be noted that the three-step path search method proposed in the above example is only an illustrative example used to illustrate the path search process. In actual applications, four-step search, two-step search, one-step search and other search steps can also be used to achieve the same purpose.

[0140] Step B: After the Nth path search is completed, the similarity between each candidate pixel point and the target pixel point in the search path is calculated.

[0141] When i=N, the computer device completes N path searches and searches for the N+1th candidate pixel point. Subsequently, the similarity between the N+1th candidate pixel point and the target pixel point in the path search is calculated.

[0142] For example, please refer to Figure 4 , which shows a schematic diagram of corresponding pixel points of a three-step path search provided by an exemplary embodiment of the present application, wherein, in the first path search, with pre_iir_y(i, j) point as the center and r=r1=6 as the radius, the points of its eight neighborhoods are found, and the eight neighborhood points and pre_iir_y(i, j) point (a total of 9 points) are used as candidate pixel points, and then the similarity between the 9 candidate pixel points and the target pixel point is calculated. If the computer device finds that pre_iir_y(l, k) is the most similar to the target pixel point among the nine candidate pixel points after calculating the similarity, the second path search is entered. The second path search uses the most similar point pre_iir_y(l, k) obtained in the first search as the starting point of the path search, narrows the search radius r=r2=3, finds its four neighborhood points, and determines the point most similar to the target pixel point as (q, r) based on the same calculation rules. The third path search is similar to the second search principle, and narrows the search radius again, r=r3=1 as the radius, and searches for the point most similar to the target pixel point as (s, t), then the coordinate point (s, t) is the best matching point.

[0143] The computer device calculates the similarity by evaluating the pixel difference between the candidate pixel and the target pixel in a certain area. The similarity calculation can be performed by a variety of methods, such as calculating the mean square error and structural similarity, to obtain the similarity between each candidate pixel and the target pixel, thereby determining the best matching candidate pixel.

[0144] In a possible implementation, the rule for calculating the similarity between the candidate point and the target pixel point in the search path is as follows:

[0145] First, the computer device calculates the absolute value of the difference between the Y channel components of each pixel in the third ROI window in the denoised image and the fourth ROI window in the intermediate image, the third ROI window is centered on the candidate pixel point, and the fourth ROI window is centered on the target pixel point.

[0146] In the calculation process, the ROI window is used to limit the calculation area to ensure that the similarity calculation is concentrated on the target pixel and its surrounding area. The size of the third ROI window and the fourth ROI window can be set according to specific needs. For example, the size of the ROI window is w×w pixels, and the window contains the target pixel and the surrounding pixels of the candidate pixel. The Y channel is the brightness component in the YUV color space. The difference in the Y channel can better reflect the structural changes and brightness changes of the image. By calculating the difference of the Y channel component, the similarity of the image can be captured more accurately.

[0147] Then, the sum of the absolute values ​​of the differences of the Y channel components of each pixel is determined as the similarity.

[0148] Among them, the pixel point with a smaller absolute value of the difference of the Y channel component in the ROI window indicates that the similarity between the corresponding pixel point in the third ROI window and the pixel point with the same coordinates in the fourth ROI window is higher, and vice versa, the pixel point with a larger absolute value of the difference of the Y channel component in the ROI window indicates that the similarity between the corresponding pixel point in the third ROI window and the pixel point with the same coordinates in the fourth ROI window is lower. After summing up the absolute values ​​of the differences of all the pixels in the ROI window, the overall similarity level of the third ROI window and the fourth ROI window is obtained, that is, the similarity between the candidate pixel point and the target pixel point, which is then used to determine whether the candidate pixel point is the best matching point.

[0149] Step C: determine the candidate pixel point in the search path that has the highest similarity with the target pixel point as the best matching point.

[0150] Through multiple path searches, the search accuracy is gradually improved, and finally the pixel point that best matches the target pixel point is selected as the best matching point. This matching point is the point used to map the pixel position between the current frame image and the previous frame image during motion compensation.

[0151] Step 302: copy pixel data in the first ROI window in the denoised image to the second ROI window in the compensation frame image.

[0152] The ROI window (region of interest window) refers to a specific area in the image, which is set around the traversed target pixel point. The first ROI window is centered on the best matching point, and the second ROI window is centered on the compensation pixel point at the target pixel point coordinates. Both the first ROI window and the second ROI window contain w×w pixels.

[0153] Since the traversal method with a row interval of w and a column interval of w is adopted when traversing the intermediate image, different first ROI windows do not contain overlapping pixels, which can ensure that the pixel data in the local area of ​​the motion compensation process is accurately copied, thereby generating a compensated frame image.

[0154] Step 303, traverse the target pixel points in the current frame image and the compensation frame pixel points in the compensation frame image.

[0155] The target pixel is each pixel in the current frame image, and the compensation frame pixel is the pixel in the previous frame image after motion compensation. The above two pixel points are traversed to compare each corresponding pixel in the current frame image and the compensation frame image to calculate the compensation error.

[0156] In a possible implementation, the computer device traverses the target pixel points in the current frame image and the compensation frame pixel points in the compensation image in a row interval of a, thereby ensuring that the calculation of the compensation error covers the entire image area and improving the accuracy of the error estimation.

[0157] Optionally, the traversal may also be performed in a column interval manner, and the traversal manner for the current frame image should be the same as the traversal manner for the compensation frame image. For example, the traversal may be performed in a manner in which the row interval is a and the column interval is b.

[0158] Step 304, calculating the sum of the absolute values ​​of the Y channel component differences between the fifth ROI window in the current frame image and the sixth ROI window in the compensation frame image.

[0159] The fifth ROI window is centered on the target pixel point, and the sixth ROI window is centered on the compensation frame pixel point. That is, the fifth and sixth ROI windows are respectively set around the target pixel point and the compensation frame pixel point to calculate the difference in pixel values ​​in the two regions.

[0160] By calculating the absolute values ​​of the pixel differences in the two ROI windows and accumulating them, a comprehensive difference value is obtained, and the comprehensive difference value can reflect the compensation error between the fifth ROI window in the current frame image and the sixth ROI window in the compensation frame image.

[0161] Step 305, calculating the mean value of the Y channel component of each pixel in the fifth ROI window, and determining the ratio of the sum of the absolute values ​​of the Y channel component differences of each pixel to the mean value as the compensation error corresponding to the target pixel.

[0162] The computer device calculates the mean value of each pixel in the ROI window, which is conducive to globally evaluating the pixel brightness characteristics in the ROI window area, thereby normalizing the local error. By calculating the mean value, the error caused by a single pixel difference can be eliminated, and a more stable and representative error evaluation indicator can be obtained.

[0163] Optionally, the Y channel components of all pixels in the ROI window are summed and divided by the number of pixels to obtain the average brightness of the area.

[0164] The ratio of the sum of the absolute values ​​of the Y channel component differences of each pixel to the mean is determined as the compensation error corresponding to the target pixel. This ratio can reflect the relative degree of brightness difference between the current frame image and the compensated frame image. The larger the ratio, the larger the compensation error, which may be caused by poor motion compensation effect. The smaller the ratio, the smaller the compensation error and the better the compensation effect.

[0165] Optionally, the pixel median, mean square, median square and other data can be used to replace the mean of each pixel in the ROI window, and the ratio of the Y channel component difference of each pixel point to one of the items is determined as the compensation error corresponding to the target pixel point.

[0166] In a possible implementation, when an interval traversal method is used, some target pixels may not be traversed. In this case, for the target pixels that have not been traversed, the compensation errors corresponding to the two adjacent target pixels in the pixel row are linearly interpolated to determine the compensation errors corresponding to the target pixels that have not been traversed.

[0167] Optionally, the computer device performs weighted averaging of compensation error differences between adjacent pixel points through linear interpolation, thereby obtaining a relatively accurate estimation value as the compensation error corresponding to the target pixel point that has not been traversed.

[0168] The use of interval traversal is beneficial to reducing the overall amount of computer calculations, and then estimating the compensation errors corresponding to the target pixels that have not been traversed through linear interpolation. It also ensures that the compensation errors of each pixel in the image are calculated, ensuring that every detail in the motion compensation process can be reasonably adjusted in preparation for the next step of calculation.

[0169] In the embodiment of the present application, motion compensation is first performed. During the motion compensation process, the compensation frame image can be accurately derived based on the similarity relationship between the intermediate image and the denoised image to ensure the continuity and consistency of the image, which is conducive to the compensation error based on motion compensation and determining a more appropriate filter coefficient, thereby improving the overall quality of the video image. In addition, the embodiment of the present application provides a gradually expanded path search method, which can effectively improve the matching accuracy, can more accurately match similar areas in the image in dynamic scenes, and gradually adjust the search radius during multiple path searches, can accurately match similar areas in the image, reduce the error of motion estimation, and improve the quality of motion compensation. In addition, similarity is calculated based on the Y channel component to reduce the matching error caused by changes in image brightness, motion or uneven illumination, and can more accurately obtain the compensation frame image, further improving the denoising effect.

[0170] 2. Estimation of Noise

[0171] For illustration, please refer to Figure 5 , which shows a flowchart of a noise estimation calculation process provided by an exemplary embodiment of the present application, the process includes the following steps:

[0172] Step 501, calculate the absolute value of the difference between the Y channel component of each pixel in the current frame image and the intermediate image, and determine it as the pixel difference.

[0173] When calculating pixel differences, the computer device compares the Y channel components of the corresponding pixels in the current frame image and the intermediate image to obtain the brightness difference of each pixel. By calculating the absolute value, it can ensure that the difference is positive and avoid deviations caused by negative differences, thereby more accurately evaluating the brightness changes between images.

[0174] Step 502, traverse each pixel point of the current frame image, calculate the variance of the pixel difference value corresponding to each pixel in different seventh ROI windows, and obtain the estimated noise corresponding to each pixel point.

[0175] The seventh ROI window is centered on the traversed pixel point.

[0176] The computer device calculates the brightness difference of all pixels in the seventh ROI window and finds the variance of these differences, thereby effectively evaluating the noise level of the area in the image.

[0177] Optionally, before performing variance calculation, the computer device performs threshold processing, enhancement processing, etc. on the detail layer, so that the variance calculation conforms to the noise characteristics of the area.

[0178] It should be noted that in the interval traversal method adopted in the above embodiments of the present application, the size of the row interval and the column interval, as well as the selected ROI window size can be adjusted according to actual conditions during the application process.

[0179] In the embodiment of the present application, the noise level of each pixel can be accurately estimated based on the above steps, thereby providing the necessary basis for subsequent denoising. The noise estimation method based on the brightness channel (Y channel component) difference and local variance can effectively capture the noise characteristics in the image.

[0180] Please refer to Figure 6 , which shows a flowchart of a video frame image denoising process provided by an exemplary embodiment of the present application.

[0181] First, the current frame image is converted from the RGB color space to the YUV space, and the Y channel component is spatially filtered, and noise estimation is performed based on the result of spatial filtering to obtain estimated noise. At the same time, motion estimation is performed based on the Y channel component of the previous frame output image (filtered image) and the Y channel component of the current frame image after spatial filtering, and then motion compensation is performed. Subsequently, the computer device determines the compensation error based on the intermediate image after spatial filtering and the compensated frame image obtained by motion compensation. Then, the filter coefficient is determined based on the compensation error and the estimated noise, and the components of the three channels of Y, U and V in the current frame image are filtered based on the filter coefficient to obtain a new filtered image, which is used as the "previous frame output image" in the denoising process of the next frame image.

[0182] Please refer to Figure 7 , which shows a block diagram of a computer device provided by an exemplary embodiment of the present application. The electronic device 700 in the present application may include one or more of the following components: a processor 710 and a memory 720.

[0183] Optionally, the processor 710 executes the steps of the video frame image denoising method provided in any of the above embodiments by running or executing instructions, programs, code sets or instruction sets stored in the memory 720, and calling data stored in the memory 720.

[0184] In addition, the processor can also perform various functions of the device and process data. Optionally, the processor 710 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 710 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the touch display; the NPU is used to implement artificial intelligence (AI) functions; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 710, but may be implemented separately through a chip.

[0185] The memory 720 may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory 720 includes a non-transitory computer-readable storage medium. The memory 720 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 720 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc.; the data storage area may store data created according to the use of the device (such as audio data, a phone book), etc.

[0186] The device in the embodiment of the present application further includes a communication component 730 and a display component 740. The communication component 730 may be a Bluetooth component, a WiFi component, an NFC (Near Field Communication) component, etc., for communicating with an external device (server or other device) via a wired or wireless network; the display component 740 is used to display a graphical user interface and / or receive user interaction operations.

[0187] Optionally, the computer device includes a computer system, which may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0188] In addition, those skilled in the art will appreciate that the structure of the device shown in the above drawings does not constitute a limitation on the device, and the device may include more or fewer components than shown, or combine certain components, or arrange the components differently. For example, the device also includes components such as a radio frequency circuit, an input unit, a sensor, an audio circuit, a speaker, and a power supply, which will not be described in detail here.

[0189] An embodiment of the present application also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the storage medium stores at least one program code, and the program code is loaded and executed by a processor to implement the video frame image denoising method as described in any of the above embodiments.

[0190] In the context of the present disclosure, a computer-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0191] The embodiment of the present application provides a computer program product, which includes computer instructions stored in a computer-readable storage medium. When the computer program is executed by a processor, the video frame image denoising method described in any of the above embodiments is implemented.

Claims

1. A video frame image denoising method, characterized in that: The method comprises: Perform spatial domain denoising on the current frame image in the color space to obtain an intermediate image; Performing motion compensation on the denoised image of the previous frame based on the intermediate image to obtain a compensated frame image; Determine a compensation error based on the compensation frame image and the current frame image; Determining estimated noise of the current frame image based on the intermediate image and the current frame image; determining filter coefficients of a time domain filter based on the estimated noise and the compensated error; The current frame image is subjected to time domain denoising based on the time domain filter to obtain a denoised image corresponding to the current frame image.

2. The method according to claim 1, characterized in that The step of determining a filter coefficient of a time domain filter based on the estimated noise and the compensation error comprises: determining an interpolation factor based on the compensation error and the estimated noise; A first filter coefficient and a second filter coefficient of the time domain filter are determined based on the interpolation factor and an adjustment parameter, wherein the adjustment parameter is used to control a balance between error compensation and noise suppression of the time domain filter.

3. The method according to claim 2, characterized in that The determining of the interpolation factor based on the compensation error and the estimated noise comprises: Determine the ratio of the compensation error to the estimated noise as the interpolation factor; The determining the first filter coefficient and the second filter coefficient of the time domain filter based on the interpolation factor and the adjustment parameter includes: Determine the first filter coefficient based on a curve formula formed by performing a power operation on the interpolation factor and the adjustment parameter; The second filter coefficient is determined based on the first filter coefficient, wherein a sum of the first filter coefficient and the second filter coefficient is 1, or a sum of the first filter coefficient and the second filter coefficient is not 1.

4. The method according to claim 3, characterized in that In the step of "determining the first filter coefficient based on a curve formula formed by performing a power operation on the interpolation factor and the adjustment parameter", the formula for determining the first filter coefficient is: In the step of "determining the second filter coefficient based on the first filter coefficient", the formula for determining the second filter coefficient is: coef2=1-coef1, or coef2=scale*(1-coef1) Among them, coef1 represents the first filter coefficient, coef2 represents the second filter coefficient, curve_th represents the adjustment parameter, scale represents the configurable hyperparameter, and I represents the interpolation factor.

5. The method according to claim 1, characterized in that The step of performing motion compensation on the denoised image of the previous frame based on the intermediate image to obtain a compensated frame image includes: Traversing each target pixel in the intermediate image in a manner of row interval w and column interval w, performing a similarity search on the denoised image based on the target pixel, and determining a best matching point of the target pixel, wherein the best matching point has the highest similarity with the best matching point; The pixel data in the first ROI window in the denoised image is copied to the second ROI window in the compensated frame image, wherein the first ROI window is centered on the best matching point, and the second ROI window is centered on the compensated pixel point at the coordinates of the target pixel point, and both the first ROI window and the second ROI window contain w×w pixels.

6. The method according to claim 5, characterized in that The performing similarity search on the denoised image based on the target pixel point to determine the best matching point of the target pixel point includes: In the denoised image, taking the first candidate pixel point at the target pixel point coordinate as the starting point, performing N path searches; After the Nth path search is completed, the similarity between each candidate pixel point in the search path and the target pixel point is calculated; The candidate pixel point in the search path that has the highest similarity to the target pixel point is determined as the best matching point.

7. The method according to claim 6, characterized in that The method of performing N path searches in the denoised image with the first candidate pixel at the target pixel coordinate as the starting point includes: In the case of the first path search, taking the first candidate pixel point as the center, searching for a second candidate pixel point on a first radius; Calculating the similarity between each second candidate pixel point in the search path and the target pixel point; In the case of performing the i-th path search, taking the i-th candidate point having the highest similarity to the target pixel point as the center, searching for the i+1-th candidate pixel point on the i-th radius; Calculate the similarity between each i+1th candidate pixel point in the search path and the target pixel point; The i-th radius of the i-th path search is greater than the i+1-th radius of the i+1-th path search.

8. The method according to any one of claims 5 or 6, characterized in that: The calculating the similarity between each candidate pixel point in the search path and the target pixel point includes: Calculate the absolute value of the difference between the Y channel component of each pixel in the third ROI window in the denoised image and the fourth ROI window in the intermediate image, wherein the third ROI window is centered on the candidate pixel point, and the fourth ROI window is centered on the target pixel point; The sum of the absolute values ​​of the differences of the Y channel components of each pixel is determined as the similarity.

9. The method according to claim 1, characterized in that: The determining of the compensation error based on the compensation frame image and the current frame image includes: Traversing the target pixel points in the current frame image and the compensation frame pixel points in the compensation frame image; Calculate the sum of the absolute values ​​of the differences between the Y channel components of each pixel point in the fifth ROI window in the current frame image and the sixth ROI window in the compensation frame image, wherein the fifth ROI window is centered on the target pixel point and the sixth ROI window is centered on the compensation frame pixel point; Calculate the mean value of the Y channel component of each pixel point in the fifth ROI window; The ratio of the sum of the absolute values ​​of the differences of the Y channel components of each pixel point to the mean value is determined as the compensation error corresponding to the target pixel point.

10. The method according to claim 9, characterized in that The traversing the target pixel points in the current frame image and the compensation frame pixel points in the compensation frame image includes: Traversing the target pixel points in the current frame image and the compensated frame pixel points in the compensated image in a row interval of a; The method further comprises: For the target pixel point that has not been traversed, linear interpolation is performed on the compensation errors corresponding to two adjacent target pixel points in the pixel row to determine the compensation error corresponding to the target pixel point that has not been traversed.

11. The method according to claim 1, characterized in that: The step of determining the estimated noise of the current frame image based on the intermediate image and the current frame image includes: Calculate the absolute value of the difference between the Y channel components of each pixel in the current frame image and the intermediate image, and determine it as the pixel difference; Each pixel point of the current frame image is traversed, and the variance of the pixel difference value corresponding to each pixel in different seventh ROI windows is calculated to obtain the estimated noise corresponding to each pixel point, and the seventh ROI window is centered on the traversed pixel point.

12. The method according to claim 1, characterized in that The performing spatial domain denoising on the current frame image in the color space to obtain an intermediate image includes: Convert the current frame image from the RGB color space to the YUV color space to obtain a Y channel component, a U channel component, and a V channel component; The Y channel component is subjected to denoising processing, and the U channel component and the V channel component are maintained to obtain the intermediate image.

13. The method according to claim 12, characterized in that The performing time domain denoising on the current frame image based on the time domain filter to obtain a denoised image corresponding to the current frame image includes: Based on the time domain filter, the Y channel component, the U channel component and the V channel component of the current frame image are filtered respectively to obtain the filtered denoised image.

14. An electronic device comprising: at least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the video frame image denoising method as described in any one of claims 1-13.

15. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the video frame image denoising method according to any one of claims 1 to 13.

16. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the video frame image denoising method according to any one of claims 1 to 13 is implemented.

Images