Video image denoising method, device, equipment and storage medium

By extracting the moving areas in the video image sequence and performing airspace denoising, combined with Wiener filtering to calculate the superimposed weight, the problem of insufficient denoising effects and 'stay' in the video denoising technology is solved, and the removal of video images is significantly improved Noise effect.

CN115242933BActive Publication Date: 2024-11-01GUANGZHOU SHIYUAN ELECTRONICS CO LTD +1
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202110437115.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-04-22
Publication Date
2024-11-01
Estimated Expiration
2041-04-22

AI Technical Summary

Technical Problem

When handling moving areas, existing video denoising technologies are prone to insufficient denoising effect and ‘stay’, especially when there are flat areas in the image, they cannot effectively distinguish between moving and stationary areas, resulting in poor video effects.

Method used

By acquiring the video image sequence, extracting the motion area, and performing airspace denoising on the motion area of ​​the reference frame image, the superposition weight of pixels at the same coordinate position in the multi-frame continuous image is calculated based on Wiener filtering, and the pixels after denoising are reasonably superimposed value to ensure effective denoising of the moving area.

Benefits of technology

It effectively solves the problem of insufficient denoising effect and 'stay' in motion areas, improves the denoising quality of video images, and significantly improves the denoising effect of the moving areas when including flat areas.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115242933B_ABST
    Figure CN115242933B_ABST
Patent Text Reader

Abstract

The present invention discloses a video image denoising method, device, equipment and storage medium. By pre-singly denoising the motion region of the reference frame image separately in the spatial domain, and calculating the superposition weights of the pixel points at the same coordinate position in multiple continuous frames of images based on Wiener filtering, and then superimposing the pixel values ​​of the pixel points of the reference frame image after spatial domain denoising with the pixel values ​​of the pixel points at the same coordinate position in other frame images in the multiple continuous frames of images according to the corresponding superposition weights, the motion region can be effectively denoised, solving the problem of insufficient denoising effect in the motion region and the "residence" problem in the motion region.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of image processing technology, and in particular to a video image denoising method, device, equipment and storage medium. Background Art

[0002] Noise is widely distributed in natural images. When the noise in an image is severe, it will seriously affect the image quality and visual effects. Denoising technology for image and video scenes has always been a core research topic. Denoising technology can be widely used in a large number of video and image scenes, such as video conferencing, video calls, taking pictures, image beautification, etc.

[0003] Different from general image denoising technology, video denoising technology needs to consider more about the use of multi-frame information and the processing of motion areas. If the effective information of multiple frames does not consider the existence of motion, assuming that the noise conforms to the Gaussian distribution, the aligned pixels can be denoised by a good multi-frame average. However, due to the existence of moving objects in the scene, after the superposition of continuous video frames, if the solution of the motion area or the superposition weight is unreasonable, there will be non-overlapping in the motion area, which is reflected in the video as ghosting.

[0004] To address this problem, most existing solutions use the method of adaptively adjusting the weight of the moving area to solve the problem of edge smearing. However, if there is a flat area in the moving area (that is, the area with the same pixel values ​​in the image), this part of the image cannot be judged as a moving area, and is therefore treated as a static area. In terms of video effect, there will be a clear separation from the moving area, and there will be a certain "residence" phenomenon. Summary of the invention

[0005] The present invention provides a video image denoising method, device, equipment and storage medium, which can effectively denoise a moving area and solve the problem of insufficient denoising effect in the moving area.

[0006] In a first aspect, an embodiment of the present invention provides a video image denoising method, comprising:

[0007] Acquire a video image sequence, wherein the video image sequence includes a plurality of continuous frames of images;

[0008] Extracting a motion region from the video image sequence;

[0009] Denoising the motion region of a reference frame image in spatial domain, wherein the reference frame image is the last frame image in the image sequence;

[0010] Calculating the superposition weights of the pixels at the same coordinate position in the multiple frames of continuous images based on Wiener filtering;

[0011] The pixel values ​​of the pixel points of the reference frame image after spatial domain denoising are superimposed with the pixel values ​​of the pixel points at the same coordinate position in other frame images in the multiple continuous frames of images according to the corresponding superposition weights to obtain a denoised video image.

[0012] In a second aspect, an embodiment of the present invention further provides a video image denoising device, comprising:

[0013] An image acquisition module, used to acquire a video image sequence, wherein the video image sequence includes a plurality of continuous frames of images;

[0014] A motion region extraction module, used for extracting a motion region from the video image sequence;

[0015] A spatial denoising module, used for spatial denoising of a motion region of a reference frame image, wherein the reference frame image is the last frame image in the image sequence;

[0016] A superposition weight calculation module, used for calculating the superposition weights of pixel points at the same coordinate position in the multiple frames of continuous images based on Wiener filtering;

[0017] The superposition module is used to superimpose the pixel values ​​of the pixel points of the reference frame image after spatial denoising with the pixel values ​​of the pixel points at the same coordinate position in other frame images in the multiple continuous frames according to the corresponding superposition weights to obtain the denoised video image.

[0018] In a third aspect, an embodiment of the present invention further provides a computer device, including:

[0019] one or more processors;

[0020] A storage device for storing one or more programs;

[0021] When the one or more programs are executed by the one or more processors, the one or more processors implement the video image denoising method provided in the first aspect of the present invention.

[0022] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the video image denoising method provided in the first aspect of the present invention.

[0023] The video image denoising method provided by the embodiment of the present invention extracts the motion region from the video image sequence after acquiring the video image sequence, performs spatial denoising on the motion region of the reference frame image, and then calculates the superposition weights of the pixels at the same coordinate position in multiple continuous frames of images based on Wiener filtering, and then superimposes the pixel values ​​of the pixels of the reference frame image after spatial denoising with the pixel values ​​of the pixels at the same coordinate position in other frame images in the multiple continuous frames of images according to the corresponding superposition weights to obtain the denoised video image. By pre-spatial denoising the motion region of the reference frame image separately, and calculating the superposition weights of the pixels at the same coordinate position in multiple continuous frames of images based on Wiener filtering, and then superimposing the pixel values ​​of the pixels of the reference frame image after spatial denoising with the pixel values ​​of the pixels at the same coordinate position in other frame images in the multiple continuous frames of images according to the corresponding superposition weights, the motion region can be effectively denoised, solving the problem of insufficient denoising effect in the motion region and the "residence" problem of the motion region. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A flowchart of a video image denoising method provided in Embodiment 1 of the present invention;

[0025] Figure 2A , 2B A flowchart of a video image denoising method provided in Embodiment 2 of the present invention;

[0026] Figure 3 A video image denoising device provided by Embodiment 3 of the present invention;

[0027] Figure 4 A schematic diagram of the structure of a computer device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0028] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0029] Embodiment 1

[0030] Figure 1 This is a flow chart of a video image denoising method provided in Embodiment 1 of the present invention. This embodiment is applicable to the case where there is a flat area in a video image. The method can be performed by a video image denoising device provided in an embodiment of the present invention. The device can be implemented in software and / or hardware and is usually configured in a computer device, such as Figure 1 As shown, the method specifically comprises the following steps:

[0031] S101: Acquire a video image sequence, where the video image sequence includes a plurality of continuous frames of images.

[0032] In the embodiment of the present invention, a video stream may be collected in real time to obtain a video image sequence therefrom. The video image sequence may include multiple frames of continuous images.

[0033] S102: Extracting a motion region from a video image sequence.

[0034] The motion area is the area where the moving object exists in the video image, which is manifested in the video image as the difference in pixel values ​​of the pixels at the same coordinate position of the two frames. In the embodiment of the present invention, after the video image sequence is acquired, the motion area is extracted from the video image sequence. Exemplarily, in some embodiments of the present invention, the optical flow method or the method of determining the difference pixel by pixel and thresholding, or other methods can be specifically used, which are not limited in the embodiment of the present invention.

[0035] S103 , performing spatial domain denoising on a motion region of a reference frame image, where the reference frame image is the last frame image in the image sequence.

[0036] In the embodiment of the present invention, the reference frame image is the last frame image in the image sequence. Denoising a frame image is also called spatial domain denoising. The essence of spatial domain denoising is the weighted superposition of pixel values ​​at multiple similar positions in the reference frame image, which is consistent with the principle of weighted superposition of pixel values ​​at different times at the same position in the time domain.

[0037] S104, calculating the superposition weights of pixels at the same coordinate position in multiple frames of continuous images based on Wiener filtering.

[0038] The images collected in real scenes usually have noise signals and image signals superimposed on each other, which can be described by an additive model:

[0039] x=s+n

[0040] Among them, x is the collected image, s is the original signal, and n is the superimposed noise signal. If the noise is too large, the original signal will be submerged in the noise, and the image x obtained will have very large noise points; if the noise is small relative to the original signal, the image quality will be very good, which is the real signal image after denoising that we want to obtain.

[0041] Based on the idea of ​​Wiener filtering, a filter is designed to act on each frame of the image to obtain the superposition weights of pixels at the same coordinate position in multiple consecutive frames of images. The expression of Wiener filtering is:

[0042]

[0043] Among them, H(z) is the superposition weight, P ss Represents the correlation coefficient between the original signal and the original signal itself, P nn represents the correlation coefficient between the noise signal and the noise signal itself, and z represents the image position coordinate.

[0044] For example, for a pixel at coordinate position (a, b), the corresponding superposition weights of each frame image are (w1, w2…w n ).

[0045] S105 , superimposing the pixel values ​​of the pixel points of the reference frame image after spatial denoising with the pixel values ​​of the pixel points at the same coordinate position in other frame images in the multiple continuous frames according to the corresponding superposition weights to obtain a denoised video image.

[0046] In an embodiment of the present invention, after obtaining the superposition weights of the pixel points of each frame image at the same coordinate position, the pixel value of the pixel point of the reference frame image at the coordinate position after spatial denoising is multiplied by the superposition weight of the reference frame image at the coordinate position, the pixel value of the pixel point of other frame images in multiple continuous frames of images at the coordinate position is multiplied by the superposition weight of the frame image at the coordinate position, and then the weighted pixel values ​​of each frame image at the coordinate position are added to obtain a denoised video image.

[0047] Since in the flat area, P of the non-moving area of ​​each frame image ss is relatively small, the superposition weight approaches 0, and the weight of each frame is almost the same, which is an average denoising effect. ss The reference frame will get a large weight, the superposition weight approaches 1, and the superposition result of the motion area is also closer to the reference frame. The final result shows that the motion area is close to the original pixel value of the reference frame after weighted superposition. This brings an obvious problem. The static area can be well denoised according to the Wiener filter formula, while the motion area basically retains the noise of the reference frame. Therefore, the embodiment of the present invention denoises the motion area of ​​the reference frame image separately in advance in the spatial domain, so that the motion area can also be denoised in the final result.

[0048] The video image denoising method provided by the embodiment of the present invention extracts the motion region from the video image sequence after acquiring the video image sequence, performs spatial denoising on the motion region of the reference frame image, and then calculates the superposition weights of the pixels at the same coordinate position in multiple continuous frames of images based on Wiener filtering, and then superimposes the pixel values ​​of the pixels of the reference frame image after spatial denoising with the pixel values ​​of the pixels at the same coordinate position in other frame images in the multiple continuous frames of images according to the corresponding superposition weights to obtain the denoised video image. By pre-spatial denoising the motion region of the reference frame image separately, and calculating the superposition weights of the pixels at the same coordinate position in multiple continuous frames of images based on Wiener filtering, and then superimposing the pixel values ​​of the pixels of the reference frame image after spatial denoising with the pixel values ​​of the pixels at the same coordinate position in other frame images in the multiple continuous frames of images according to the corresponding superposition weights, the motion region can be effectively denoised, solving the problem of insufficient denoising effect in the motion region and the "residence" problem of the motion region.

[0049] Embodiment 2

[0050] Figure 2A , 2B This is a flowchart of a video image denoising method provided in Embodiment 2 of the present invention. This embodiment is a refinement of Embodiment 1 above, and describes in detail the specific process of each step in the above embodiment, such as Figure 2A , 2B As shown, the method specifically comprises the following steps:

[0051] S201: Acquire a video image sequence, where the video image sequence includes a plurality of continuous frames of images.

[0052] In the embodiment of the present invention, a video stream may be collected in real time to obtain a video image sequence therefrom. The video image sequence may include multiple frames of continuous images.

[0053] For image processing of motion scenes, solving the motion region is a very critical step. There are many different methods for solving the motion region in terms of effect and time consumption, and it is difficult to have a method that can combine time consumption and effect. The traditional optical flow method has a more accurate solution result, but it is very time-consuming; another very simple method is to judge the difference pixel by pixel and threshold it, which can achieve a fast solution, but it is basically impossible to solve the flat area, and it is greatly affected by noise fluctuations. The following will describe an exemplary process of extracting a motion region from a video image sequence in an embodiment of the present invention, referring to steps S202-S205.

[0054] S202: Select a first neighborhood centered on each pixel point on the reference frame image.

[0055] Traverse all pixels on the reference frame image and select a first neighborhood centered on each pixel on the reference frame image. Exemplarily, the first neighborhood may be a rectangular area centered on each pixel on the reference frame image and having a size of 5×5.

[0056] S203: Calculate the similarity between the target area having the same coordinate position as the first neighborhood in other frame images and the first neighborhood.

[0057] Specifically, according to the position of the center of the first neighborhood and the size of the first neighborhood, a target area in other frame images having the same coordinate position and size as the first neighborhood is determined, and then the similarity between the first neighborhood and the corresponding target area is calculated.

[0058] Specifically, the normalized correlation coefficient between the first neighborhood and the corresponding target area can be calculated as the similarity. The calculation formula of the normalized correlation coefficient is as follows:

[0059]

[0060] Among them, ρ is the normalized correlation coefficient between the first neighborhood and the corresponding target area, x i is the pixel value of the i-th pixel in the first neighborhood, y i is the pixel value of the i-th pixel in the target area, u x is the average value of all pixel values ​​in the first neighborhood, u y is the average value of all pixel values ​​in the target area, σ x is the standard deviation of the pixel values ​​of all pixels in the first neighborhood, σ y is the standard deviation of the pixel values ​​of all pixels in the target area.

[0061] S204: Determine whether the similarity between the target area in other frame images and the first neighborhood is less than a similarity threshold.

[0062] The similarity between the target area in other frame images and the first neighborhood is compared with a preset similarity threshold to determine whether the similarity between the target area in other frame images and the first neighborhood is less than the similarity threshold. Exemplarily, the normalized correlation coefficient between the target area in other frame images and the first neighborhood is compared with a preset correlation coefficient threshold to determine whether the normalized correlation coefficient between the target area in other frame images and the first neighborhood is less than the correlation coefficient threshold.

[0063] S205 . When the similarity between the target area of ​​any frame image in the other frame images and the first neighborhood is lower than a similarity threshold, determine that the first neighborhood belongs to a motion area.

[0064] When the similarity between the target area of ​​any frame image in the other frame images and the corresponding first neighborhood is lower than the similarity threshold, it is determined that the first neighborhood belongs to the motion area.

[0065] The entire motion region is obtained by calculating the similarity between each first neighboring region and the corresponding target region.

[0066] In some embodiments of the present invention, before selecting the first neighborhood centered on each pixel point on the reference frame image, multiple frames of continuous images can be downsampled to obtain a downsampled image after downsampling, and then the motion area is extracted from the downsampled image, and then upsampled to obtain a motion area with the same resolution as the reference frame image. In this way, the extraction speed of the motion area can be improved to meet the real-time noise reduction requirements. In addition, the robustness of the motion area solution on a small scale is better, which greatly improves the situation where the flat area of ​​the motion cannot be judged, and further eliminates the "residence" problem of the motion area.

[0067] The following will describe an exemplary process of spatial denoising of a motion region of a reference frame image. In an embodiment of the present invention, a method of superimposing similar pixels based on neighborhood gradient directions is used for denoising, see steps S206-S210.

[0068] S206: Select a second neighborhood centered on each pixel point in the motion area.

[0069] Specifically, traverse each pixel point on the motion area, and select a second neighborhood of a preset size (e.g., 5×5) with each pixel point as the center. Exemplarily, after obtaining the motion area in the above steps, generate a binary mask of the same size as the reference image, wherein the value of the element in the motion area in the mask is 1, and the value of the element in the non-motion area is 0. When selecting the second neighborhood, based on the mask, select the pixel point on the reference frame image corresponding to the element with a value of 1 on the mask as the center.

[0070] S207: Calculate the gradient direction of the second neighborhood.

[0071] Specifically, the Laplacian operator is used to solve the gradient direction in the current second neighborhood. In other embodiments of the present invention, the gradient direction of the second neighborhood can also be calculated by other methods, which will not be described in detail in this embodiment of the present invention. The gradient direction can be horizontal, vertical or diagonal.

[0072] S208. Select multiple pixel points in the gradient direction as target pixel points.

[0073] Specifically, along the calculated gradient direction, multiple pixel points (for example, 5) in the gradient direction are taken as target pixel points.

[0074] S209, calculating weights of multiple target pixels based on Wiener filtering.

[0075] Specifically, the correlation coefficients of the pixel values ​​of the multiple target pixels are calculated as P in the Wiener filter formula in the above embodiment. ss Item, P nn The term remains unchanged, and the correlation coefficient between the noise signal and the noise signal itself is still used as P nn Item. ss Item and P nn Substitute the term into the Wiener filter formula to obtain the weights of multiple target pixels.

[0076] The above step S209 may include the following sub-steps:

[0077] S2091. Calculate the variance of the pixel values ​​of multiple target pixels as a first variance.

[0078] Specifically, the variance of the pixel values ​​of the plurality of target pixel points is calculated as the first variance, and the first variance is used as the correlation coefficient of the pixel values ​​of the plurality of target pixel points.

[0079] S2092, calculating the variance of the pixel values ​​of the pixel points with the same coordinate position in the flat area of ​​the multiple frames of continuous images as the second variance.

[0080] Specifically, the Sobel operator can be used to extract flat areas from multiple continuous frames of images to obtain the flat areas in each frame of the image, and then the intersection of multiple flat areas is taken as the final flat area. Then, the variance of the pixel values ​​of the pixel points with the same coordinate position in the intersection of the flat areas of the multiple continuous frames of the image is calculated as the second variance.

[0081] Generally, we believe that the real signal in the flat area is a constant, and the fluctuation of image pixels is only caused by noise. Therefore, the second variance can be used as the correlation coefficient P between the noise signal and the noise signal itself. nn .

[0082] S2094. Calculate the sum of the first variance and the second variance as a first sum value.

[0083] Substitute the first variance and the second variance into the Wiener filter formula, and calculate the sum of the first variance and the second variance as a first sum value.

[0084] S2095. Calculate the quotient of the first variance and the first sum as the weights of the multiple target pixel points.

[0085] Substitute the first variance and the second variance into the Wiener filter formula, and calculate the quotient of the first variance and the first sum as the weights of the multiple target pixels.

[0086] S210 , summing up the pixel values ​​of the plurality of target pixels according to corresponding weights, and taking the sum as the pixel value of the central pixel of the second neighborhood.

[0087] Specifically, the pixel values ​​of the multiple target pixels are multiplied by the weights, and the sum is added, and the obtained pixel value is used as the pixel value of the pixel at the center of the second neighborhood.

[0088] By performing the above denoising process on the second neighborhood of each pixel point in the motion region of the reference frame image, denoising of the motion region of the reference frame image is completed.

[0089] In another embodiment of the present invention, the motion region of the reference frame image can also be denoised in the spatial domain by using the non-local mean method. The basic idea of ​​the non-local mean method is that the estimated value of the current pixel is obtained by weighted averaging the pixels in the image that have a similar neighborhood structure to it. However, the method of similar pixel superposition based on the neighborhood gradient direction provided in the embodiment of the present invention is more efficient than the non-local mean method, and the effect is similar to that of the non-local mean method.

[0090] The following describes an exemplary process of calculating the superposition weights of pixels at the same coordinate position in multiple frames of continuous images based on Wiener filtering, with reference to steps S211-S214.

[0091] S211, calculating the variance of pixel values ​​of pixel points at the same coordinate position in multiple frames of continuous images as a first coefficient.

[0092] Calculate the variance of the pixel values ​​of the pixel points at the same coordinate position in multiple frames of continuous images. For example, calculate the variance of the pixel values ​​of the pixel points at the same coordinate position in 5 frames of images as the first coefficient, and use the first coefficient as P in the Wiener filter formula ss item.

[0093] S212, calculating the variance of the pixel values ​​of the pixel points with the same coordinate position in the flat area of ​​the multiple frames of continuous images as the second coefficient.

[0094] For example, the variance of the pixel values ​​of the pixels at the same coordinate position in the flat area of ​​the multiple consecutive frames of images has been obtained in the aforementioned step S2092, and this step only needs to be called. The second coefficient (or the second variance) is used as P in the Wiener filter formula nn item.

[0095] S213. Calculate the sum of the first coefficient and the second coefficient as a second sum value.

[0096] Substitute the first coefficient and the second coefficient into the Wiener filter formula, and calculate the sum of the first coefficient and the second coefficient as the second sum value.

[0097] S214: Calculate the quotient of the first coefficient and the second sum to obtain a superposition weight.

[0098] Substitute the first coefficient and the second coefficient into the Wiener filter formula, and calculate the quotient of the first coefficient and the second sum as the superposition weight.

[0099] S215, superimposing the pixel values ​​of the pixel points of the reference frame image after spatial denoising with the pixel values ​​of the pixel points at the same coordinate position in other frame images in the multiple continuous frames according to the corresponding superposition weights to obtain a denoised video image.

[0100] After obtaining the superposition weights of the pixels at the same coordinate position of each frame image, the pixel value of the pixel at the coordinate position of the reference frame image after spatial denoising is multiplied by the superposition weight of the reference frame image at the coordinate position, the pixel value of the pixel at the coordinate position of other frame images in multiple continuous frames of images is multiplied by the superposition weight of the frame image at the coordinate position, and then the weighted pixel values ​​of each frame image at the coordinate position are added to obtain a denoised video image.

[0101] The video image denoising method provided by the embodiment of the present invention can effectively denoise the moving area by pre-spatial denoising the moving area of ​​the reference frame image separately, and calculating the superposition weights of the pixel points at the same coordinate position in multiple continuous frames of images based on Wiener filtering, and then superimposing the pixel values ​​of the pixel points of the reference frame image after spatial denoising with the pixel values ​​of the pixel points at the same coordinate position in other frame images in the multiple continuous frames of images according to the corresponding superposition weights, thereby solving the problem of insufficient denoising effect in the moving area and the "residence" problem of the moving area. In addition, the extraction speed of the moving area can be improved to meet the real-time noise reduction requirements, improve the situation where the moving flat area cannot be judged, and further eliminate the "residence" problem of the moving area.

[0102] Embodiment 3

[0103] Figure 3 A video image denoising device provided in Embodiment 3 of the present invention includes:

[0104] An image acquisition module 301 is used to acquire a video image sequence, wherein the video image sequence includes a plurality of continuous frames of images;

[0105] A motion region extraction module 302, used to extract a motion region from the video image sequence;

[0106] A spatial denoising module 303, configured to perform spatial denoising on a motion region of a reference frame image, wherein the reference frame image is the last frame image in the image sequence;

[0107] A superposition weight calculation module 304 is used to calculate the superposition weights of pixels at the same coordinate position in multiple frames of continuous images based on Wiener filtering;

[0108] The superposition module 305 is used to superimpose the pixel values ​​of the pixel points of the reference frame image after spatial denoising with the pixel values ​​of the pixel points at the same coordinate position in other frame images in multiple consecutive frames according to the corresponding superposition weights to obtain a denoised video image.

[0109] In some embodiments of the present invention, the motion region extraction module 302 includes:

[0110] A first neighborhood selection submodule, used for selecting a first neighborhood centered on each pixel point on the reference frame image;

[0111] A similarity calculation submodule, used to calculate the similarity between the target area having the same coordinate position as the first neighborhood in other frame images and the first neighborhood;

[0112] A judgment submodule, used to judge whether the similarity between the target area in other frame images and the first neighborhood is less than a similarity threshold;

[0113] The motion region determination submodule is used to determine that the first neighborhood belongs to the motion region when the similarity between the target region of any frame image in other frame images and the first neighborhood is lower than a similarity threshold.

[0114] In some embodiments of the present invention, the video image denoising apparatus further includes:

[0115] The down-sampling module is used to perform down-sampling processing on multiple frames of continuous images before selecting a first neighborhood centered on each pixel point on the reference frame image to obtain a down-sampled image after the down-sampling processing.

[0116] In some embodiments of the present invention, the video image denoising apparatus further includes:

[0117] The up-sampling module is used to perform up-sampling processing on the motion region after determining that the first neighborhood belongs to the motion region, so as to obtain a motion region with the same resolution as the reference frame image.

[0118] In some embodiments of the present invention, the spatial domain denoising module 303 includes:

[0119] A second neighborhood selection submodule, used for selecting a second neighborhood centered on each pixel point on the motion area;

[0120] A gradient direction calculation submodule, used to calculate the gradient direction of the second neighborhood;

[0121] A target pixel point determination submodule is used to select a plurality of pixel points as target pixel points in the gradient direction;

[0122] A weight calculation submodule, used for calculating the weights of a plurality of target pixels based on Wiener filtering;

[0123] A summing submodule is used to sum the pixel values ​​of the plurality of target pixel points according to corresponding weights as the pixel value of the pixel point at the center of the second neighborhood.

[0124] In some embodiments of the present invention, the weight calculation submodule includes:

[0125] A first variance calculation unit, used to calculate the variance of the pixel values ​​of the plurality of target pixels as a first variance;

[0126] A second variance calculation unit, used to calculate the variance of pixel values ​​of pixel points with the same coordinate position in a flat area of ​​a plurality of continuous frames of images as a second variance;

[0127] A first sum calculation unit, configured to calculate a sum of the first variance and the second variance as a first sum;

[0128] A weight calculation unit is used to calculate the quotient of the first variance and the first sum as the weights of the plurality of target pixels.

[0129] In some embodiments of the present invention, the second variance calculation unit includes:

[0130] A flat region extraction subunit is used to extract flat regions from multiple consecutive frames of images using a Sobel operator;

[0131] An intersection determination subunit is used to obtain the intersection of multiple flat areas;

[0132] The second variance calculation subunit is used to calculate the variance of pixel values ​​of pixel points with the same coordinate position in the intersection of the flat areas of the multiple frames of continuous images as the second variance.

[0133] In some embodiments of the present invention, the superposition weight calculation module includes:

[0134] A first coefficient calculation submodule, used to calculate the variance of pixel values ​​of pixel points at the same coordinate position in multiple frames of continuous images as a first coefficient;

[0135] A second coefficient calculation submodule, used to calculate the variance of pixel values ​​of pixel points with the same coordinate position in a flat area of ​​multiple consecutive frames of images as a second coefficient;

[0136] A second sum calculation submodule, used for calculating the sum of the first coefficient and the second coefficient as a second sum;

[0137] The superposition weight calculation submodule is used to calculate the quotient of the first coefficient and the second sum value to obtain the superposition weight.

[0138] The above-mentioned video image denoising device can execute the video image denoising method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0139] Embodiment 4

[0140] Embodiment 4 of the present invention provides a computer device, Figure 4 A schematic diagram of the structure of a computer device provided in Embodiment 4 of the present invention is shown in FIG. Figure 4 As shown, the computer device includes:

[0141] Processor 401, memory 402, communication module 403, input device 404 and output device 405; the number of processors 401 in the computer device can be one or more. Figure 4 A processor 401 is taken as an example; the processor 401, memory 402, communication module 403, input device 404 and output device 405 in the computer device can be connected by a bus or other means. Figure 4 The above processor 401, memory 402, communication module 403, input device 404 and output device 405 may be integrated in a computer device.

[0142] The memory 402 is a computer-readable storage medium that can be used to store software programs, computer executable programs and modules, such as the module corresponding to the video image denoising method in the above embodiment. The processor 401 executes various functional applications and data processing of the computer device by running the software programs, instructions and modules stored in the memory 402, that is, implementing the above video image denoising method.

[0143] The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the microcomputer, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 402 may further include a memory remotely arranged relative to the processor 401, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0144] The communication module 403 is used to establish a connection with an external device (such as a smart terminal) and realize data interaction with the external device. The input device 404 can be used to receive input digital or character information and generate key signal input related to user settings and function control of the computer device.

[0145] The computer device provided in this embodiment can execute the video image denoising method provided in any of the above embodiments of the present invention, and has corresponding functions and beneficial effects.

[0146] Embodiment 5

[0147] Embodiment 5 of the present invention provides a storage medium containing computer executable instructions, on which a computer program is stored. When the program is executed by a processor, a video image denoising method as provided in any of the above embodiments of the present invention is implemented. The method includes:

[0148] Acquire a video image sequence, wherein the video image sequence includes a plurality of continuous frames of images;

[0149] Extracting a motion region from the video image sequence;

[0150] Denoising the motion region of a reference frame image in spatial domain, wherein the reference frame image is the last frame image in the image sequence;

[0151] Calculate the superposition weights of pixels at the same coordinate position in multiple frames of continuous images based on Wiener filtering;

[0152] The pixel values ​​of the pixels of the reference frame image after spatial domain denoising are superimposed with the pixel values ​​of the pixels at the same coordinate position in other frame images in multiple consecutive frames according to the corresponding superposition weights to obtain a denoised video image.

[0153] It should be noted that, for the device, equipment and storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0154] Through the above description of the implementation method, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a robot, a personal computer, a server, or a network device, etc.) to perform the video image denoising method described in any embodiment of the present invention.

[0155] It is worth noting that in the above-mentioned device, the various modules, sub-modules, units, and sub-units included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0156] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution device. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0157] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0158] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A video image denoising method, characterized in that: include: Acquire a video image sequence, wherein the video image sequence includes a plurality of continuous frames of images; Extracting a motion region from the video image sequence; Denoising the motion region of a reference frame image in spatial domain, wherein the reference frame image is the last frame image in the image sequence; Calculating the superposition weights of the pixels at the same coordinate position in the multiple frames of continuous images based on Wiener filtering; Superimposing the pixel values ​​of the pixel points of the reference frame image after spatial domain denoising with the pixel values ​​of the pixel points at the same coordinate position in other frame images in the plurality of continuous frames according to the corresponding superposition weights to obtain a denoised video image; The calculating the superposition weights of the pixels at the same coordinate position in the multiple frames of continuous images based on Wiener filtering includes: Calculating the variance of pixel values ​​of pixel points at the same coordinate position in the plurality of consecutive frames of images as a first coefficient; Calculate the variance of pixel values ​​of pixel points with the same coordinate position in the flat area of ​​the plurality of continuous frames of images as a second coefficient; calculating a sum of the first coefficient and the second coefficient as a second sum value; A quotient of the first coefficient and the second sum is calculated to obtain a superposition weight.

2. The video image denoising method according to claim 1, characterized in that: Extracting a motion region from the video image sequence comprises: Selecting a first neighborhood centered on each pixel point on the reference frame image; Calculating the similarity between the target area having the same coordinate position as the first neighborhood in other frame images and the first neighborhood; Determine whether the similarity between the target area in other frame images and the first neighborhood is less than a similarity threshold; When the similarity between the target area of ​​any frame image in other frame images and the first neighborhood is lower than a similarity threshold, it is determined that the first neighborhood belongs to a motion area.

3. The video image denoising method according to claim 2, characterized in that: Before selecting a first neighborhood centered on each pixel point on the reference frame image, the method further includes: Downsampling is performed on the multiple frames of continuous images to obtain downsampled images after the downsampling.

4. The video image denoising method according to claim 3, characterized in that: After determining that the first neighborhood belongs to a motion area, the method further includes: The motion region is up-sampled to obtain a motion region with the same resolution as the reference frame image.

5. The video image denoising method according to any one of claims 1 to 4, characterized in that: Spatial denoising of the motion region of the reference frame image, including: Selecting a second neighborhood centered on each pixel point on the motion area; Calculating the gradient direction of the second neighborhood; Selecting a plurality of pixels in the gradient direction as target pixels; Calculate the weights of the plurality of target pixels based on Wiener filtering; The pixel values ​​of the plurality of target pixels are summed according to corresponding weights to obtain the summed value as the pixel value of the center pixel of the second neighborhood.

6. The video image denoising method according to claim 5, characterized in that: Calculating the weights of the plurality of target pixels based on Wiener filtering includes: Calculating the variance of the pixel values ​​of the plurality of target pixels as a first variance; Calculate the variance of pixel values ​​of pixel points with the same coordinate position in the flat area of ​​the plurality of continuous frames of images as a second variance; calculating a sum of the first variance and the second variance as a first sum value; A quotient of the first variance and the first sum is calculated as weights of the plurality of target pixel points.

7. The video image denoising method according to claim 6, characterized in that: Calculating the variance of pixel values ​​of pixel points with the same coordinate position in the flat area of ​​the plurality of continuous frames of images as a second variance comprises: Using the Sobel operator to extract flat areas from the multiple consecutive frames of images; Take the intersection of multiple flat areas; The variance of the pixel values ​​of the pixel points with the same coordinate position in the intersection of the flat areas of the multiple frames of continuous images is calculated as the second variance.

8. A video image denoising device, characterized in that: include: An image acquisition module, used to acquire a video image sequence, wherein the video image sequence includes a plurality of continuous frames of images; A motion region extraction module, used for extracting a motion region from the video image sequence; A spatial denoising module, used for spatial denoising of a motion region of a reference frame image, wherein the reference frame image is the last frame image in the image sequence; A superposition weight calculation module, used for calculating the superposition weights of pixel points at the same coordinate position in the multiple frames of continuous images based on Wiener filtering; A superposition module, used for superimposing the pixel values ​​of the pixel points of the reference frame image after spatial domain denoising with the pixel values ​​of the pixel points at the same coordinate position in other frame images in the multiple continuous frames of images according to corresponding superposition weights to obtain a denoised video image; The superposition weight calculation module includes: A first coefficient calculation submodule, used to calculate the variance of pixel values ​​of pixel points at the same coordinate position in the plurality of consecutive frames of images as a first coefficient; A second coefficient calculation submodule, used for calculating the variance of pixel values ​​of pixel points with the same coordinate position in the flat area of ​​the plurality of continuous frames of images as a second coefficient; A second sum calculation submodule, used for calculating the sum of the first coefficient and the second coefficient as a second sum; The superposition weight calculation submodule is used to calculate the quotient of the first coefficient and the second sum value to obtain the superposition weight.

9. A computer device, characterized in that: include: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the video image denoising method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the video image denoising method as described in any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Inter-frame noise reduction method based on motion detection

    CN105472204A

  • Advanced interpolation techniques for motion compensation in video coding

    EP3454560A1

  • Image processing device, image processing method, and image processing program

    WO2010064316A1