Video denoising method, system and readable storage medium

Abstract

The present application relates to the technical field of video image processing, and particularly relates to a video denoising method, system and readable storage medium, the method comprising: inputting YUV video data of continuous frames; transforming the YUV video data into first-level coarse-grained data and two-level fine-grained data which are not correlated with each other; respectively performing denoising on the first-level coarse-grained data and the two-level fine-grained data; and reconstructing the first-level coarse-grained information and the second-level fine-grained information after denoising into YUV video data with correlation through inverse transformation and outputting. The YUV video data is divided into first-level coarse-grained channel data and two-level fine-grained channel data through the method of de-correlation, so that each channel is independent of each other, the correlation of noise between channels is avoided, and then the first-level coarse-grained data and the two-level fine-grained data are respectively denoised, which can effectively improve the denoising level of each channel of the video frame.

Description

Technical Field

[0001] This invention relates to the field of video image processing technology, and in particular to a video denoising method, system, and readable storage medium. Background Technology

[0002] Video data can generate noise at many stages, such as sensor signal acquisition and signal transmission. Video denoising is a fundamental task in signal processing, and its purpose is to reconstruct a clean video frame sequence from a noisy video frame sequence.

[0003] Most current denoising methods are designed for RGB image data. However, raw images obtained directly from image sensors contain rich noise information, and their noise models are relatively simple. But after image signal processing into video, the noise model becomes complex, and the correlation between different channels makes noisy video frame sequences difficult to process. Current mature video denoising methods mainly utilize searching for similar patches in the temporal and spatial dimensions. However, this method mixes spatiotemporal information without considering spatial noise distribution and temporal deformation information. Summary of the Invention

[0004] The purpose of this invention is to provide a video denoising method, system, and readable storage medium, which at least solves one of the technical problems existing in the current denoising methods.

[0005] To achieve the above objectives, the present invention provides a video denoising method, comprising:

[0006] S100, Input continuous frames of YUV video data;

[0007] S200: Transform the YUV video data into uncorrelated first-level coarse-grained data and two-level fine-grained data;

[0008] S300, Denoise the first-level coarse-grained data and the two-level fine-grained data respectively;

[0009] S400: The denoised first-level coarse-grained information and second-level fine-grained information are reconstructed into correlated YUV video data through inverse transformation and then output.

[0010] Optionally, S200 specifically includes:

[0011] The principal component analysis method is used to approximate the decorrelation transformation matrix, and the calculation formula is as follows:

[0012] I=1.0010×Yin+0.3102×Uin-0.0068×Vin

[0013] C=-0.0020×Yin-1.4125×Uin-1.5345×Vin

[0014] K=0.0000×Yin-1.0165×Uin-0.7605×Vin

[0015] Wherein, Yin, Uin, and Vin are the matrices corresponding to the YUV video data in the Y, U, and V channels, respectively; I is the coarse-grained matrix obtained after decorrelation transformation; C is the first-level fine-grained matrix obtained after decorrelation transformation; and K is the second-level fine-grained matrix obtained after decorrelation transformation.

[0016] Optionally, the matrix size corresponding to the Y, U, and V channels of the YUV video data is the same.

[0017] Optionally, in S300, a two-stage coarse-grained denoising framework is used to denoise the first-level coarse-grained data.

[0018] Optionally, in step S300, denoising the first-level coarse-grained data specifically includes:

[0019] S301. Perform spatial feature decomposition on the first-level coarse-grained data to obtain four-channel high-level spatial information and four-channel low-level spatial information.

[0020] S302. The motion features of the first-level coarse-grained data in the low-level space are approximated by using the variance similarity and matching similarity between the target frame and the average frame in the low-level space.

[0021] S303. Map the motion representation according to the low-level motion characteristics to obtain clean low-level spatial information;

[0022] S304. Construct a low-level residual correction structure and reconstruct the high-level spatial information of the target frame after denoising based on the four-channel low-level spatial information.

[0023] S305. The high-level motion features of the first-level coarse-grained data in the high-level space are approximated by using the variance similarity and matching similarity between the target frame and the average frame in the high-level space.

[0024] S306. Map the motion representation according to the high-level motion characteristics to obtain clean high-level spatial information;

[0025] S307. Construct a high-level residual correction structure and reconstruct the first-level coarse-grained data of the target frame after denoising based on the four-channel high-level spatial information.

[0026] Optionally, in S300, a similarity denoising framework is used to denoise the two-level fine-grained data.

[0027] Optionally, in step S300, denoising the two-level fine-grained data specifically includes:

[0028] S308. Perform local similarity calculation in the time domain on the secondary fine-grained data;

[0029] S309. Adaptively calculate the denoising intensity based on local similarity information;

[0030] S310. Perform adaptive similarity denoising on the secondary fine-grained level according to the denoising intensity.

[0031] Based on the same technical concept, the present invention also provides a video denoising system, comprising:

[0032] The video data input module is configured to input YUV video data in consecutive frames;

[0033] The channel decorrelation transformation module is configured to transform the YUV video data into uncorrelated first-level coarse-grained data and two-level fine-grained data.

[0034] The denoising module is configured to denoise both the first-level coarse-grained data and the two-level fine-grained data.

[0035] The channel decorrelation inverse transform module is configured to reconstruct the denoised first-level coarse-grained information and second-level fine-grained information into correlated YUV video data for output through inverse transform.

[0036] Optionally, the denoising module includes a coarse-grained denoising unit and a fine-grained denoising unit. The coarse-grained denoising unit is configured to use a two-stage coarse-grained denoising framework to denoise the first-level coarse-grained data, and the fine-grained denoising unit is configured to use a similarity denoising framework to denoise the two-level fine-grained data.

[0037] Based on the same technical concept, the present invention also provides a readable storage medium having a computer program stored thereon, which, when executed, can implement the video denoising method described above.

[0038] The video denoising method, system, and readable storage medium provided by this invention have at least one of the following beneficial effects:

[0039] 1) By using a decorrelation method, the YUV video data is divided into a first-level coarse-grained channel data and a second-level fine-grained channel data, making each channel independent of each other and avoiding the influence of noise correlation between channels. Then, the first-level coarse-grained data and the second-level fine-grained data are denoised separately, which can effectively improve the denoising level of each channel of the video frame.

[0040] 2) By employing a two-stage coarse-grained denoising framework to denoise the first-level coarse-grained data, dynamic features are extracted from multiple video frames. Utilizing the dynamic feature information of adjacent frames, high-quality, clean coarse-grained video frames are gradually obtained from lower to higher spatial layers, effectively improving the denoising level of the video frame's coarse-grained channel. Furthermore, introducing a two-stage residual structure during the coarse-grained data denoising process effectively preserves video detail information, further enhancing the overall video quality. Attached Figure Description

[0041] Those skilled in the art will understand that the accompanying drawings are provided to better understand the invention and do not constitute any limitation on the scope of the invention. Wherein:

[0042] Figure 1 This is a flowchart illustrating the steps of a video denoising method provided in an embodiment of the present invention;

[0043] Figure 2 A flowchart of a video denoising method provided in an embodiment of the present invention;

[0044] Figure 3 This is a flowchart of denoising primary coarse-grained data provided in an embodiment of the present invention;

[0045] Figure 4 This is a flowchart of denoising secondary fine-grained data provided in an embodiment of the present invention;

[0046] Figure 5 This is a structural block diagram of a video denoising system provided in an embodiment of the present invention.

[0047] In the attached image:

[0048] 100 - Video data input module; 200 - Channel decorrelation transformation module; 300 - Denoising module; 301 - Spatial feature decomposition calculation subunit; 302 - Low-level spatial motion feature calculation subunit; 303 - Low-level spatial motion representation mapping subunit; 304 - Low-level spatial residual correction subunit; 305 - High-level spatial motion feature calculation subunit; 306 - High-level spatial motion representation mapping subunit; 307 - High-level spatial residual correction subunit; 308 - Local similarity calculation subunit; 309 - Adaptive denoising intensity calculation subunit; 310 - Adaptive similarity denoising subunit; 400 - Channel decorrelation inverse transformation module. Detailed Implementation

[0049] To make the objectives, advantages, and features of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the drawings are in a very simplified form and use non-precise proportions, and are only used to facilitate and clearly illustrate the purpose of the embodiments of this invention. Please refer to the accompanying drawings to make the objectives, features, and advantages of this invention more apparent and understandable. It should be understood that the structures, proportions, sizes, etc., depicted in the accompanying drawings are only used to complement the content disclosed in the specification, for those skilled in the art to understand and read, and are not intended to limit the implementation conditions of this invention. Any modifications to the structure, changes in proportions, or adjustments to the size, if they are the same as or similar to the effects and objectives achieved by this invention, should still fall within the scope of the technical content disclosed in this invention.

[0050] As used herein, the singular forms “a,” “an,” and “the” include plural objects unless otherwise expressly indicated. As used herein, the term “or” is generally used to include “and / or” unless otherwise expressly indicated. As used herein, the term “a number” is generally used to include “at least one” unless otherwise expressly indicated. As used herein, the term “at least two” is generally used to include “two or more” unless otherwise expressly indicated. Furthermore, the terms “first,” “second,” and “third” are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as “first,” “second,” or “third” may explicitly or implicitly include one or at least two of that feature.

[0051] In the description of this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0052] Please refer to Figures 1-2 This invention provides a video denoising method, comprising the following steps:

[0053] S100, Input continuous frames of YUV video data;

[0054] S200: Transform the YUV video data into uncorrelated first-level coarse-grained data and two-level fine-grained data;

[0055] S300, Denoise the first-level coarse-grained data and the two-level fine-grained data respectively;

[0056] S400: The denoised first-level coarse-grained information and second-level fine-grained information are reconstructed into correlated YUV video data through inverse transformation and then output.

[0057] By using a decorrelation method, the YUV video data is divided into a first-level coarse-grained channel data and a second-level fine-grained channel data, making each channel independent of each other and avoiding the influence of noise correlation between channels. Then, the first-level coarse-grained data and the second-level fine-grained data are denoised separately, which can effectively improve the denoising level of each channel of the video frame.

[0058] First, execute S100 to input consecutive frames of YUV video data.

[0059] Next, step S200 is executed to transform the YUV video data into uncorrelated first-level coarse-grained data and two-level fine-grained data. This embodiment uses principal component analysis to approximate an decorrelation transformation matrix, transforming the YUV video data into uncorrelated first-level coarse-grained data and two-level fine-grained data. The specific calculation formula is as follows:

[0060]

[0061] Wherein, Yin, Uin, and Vin are the matrices corresponding to the YUV video data in the Y, U, and V channels, respectively; I is the coarse-grained matrix obtained after decorrelation transformation; C is the first-level fine-grained matrix obtained after decorrelation transformation; and K is the second-level fine-grained matrix obtained after decorrelation transformation.

[0062] The YUV video data uses the default YUV mode, where the Y, U, and V channels have the same matrix size. For data in other YUV modes such as YUV422 or YUV420, the UV channel data size needs to be adjusted to match the Y channel data before processing. There are already mature processing solutions for this process in the existing technology, so this valve will not go into detail.

[0063] Then, S300 is executed to denoise the first-level coarse-grained data and the two-level fine-grained data respectively.

[0064] In this embodiment, a two-stage coarse-grained denoising framework is used to denoise the first-level coarse-grained data. For details, please refer to [link / reference needed]. Figure 3 The denoising process for the first-level coarse-grained data includes:

[0065] S301. Perform spatial feature decomposition on the first-level coarse-grained data to obtain four-channel high-level spatial information and four-channel low-level spatial information.

[0066] S302. The variance similarity and matching similarity between the target frame and the average frame in the low-level space are used to approximate the low-level motion features to obtain the motion features of the first-level coarse-grained data in the low-level space.

[0067] S303. Map the motion representation according to the low-level motion characteristics to obtain clean low-level spatial information;

[0068] S304. Construct a low-level residual correction structure and reconstruct the high-level spatial information of the target frame after denoising based on the four-channel low-level spatial information.

[0069] S305. The variance similarity and matching similarity between the target frame and the average frame in the high-level space are used to approximate the high-level motion features of the first-level coarse-grained data to obtain the high-level motion features of the first-level coarse-grained data in the high-level space.

[0070] S306. Map the motion representation according to the high-level motion characteristics to obtain clean high-level spatial information;

[0071] S307. Construct a high-level residual correction structure and reconstruct the first-level coarse-grained data of the target frame after denoising based on the four-channel high-level spatial information.

[0072] First, execute S301 to perform spatial feature decomposition on the first-level coarse-grained data, obtaining four-channel high-level spatial information and four-channel low-level spatial information. Assume {I i-n ,…,I i-1 ,I i ,I i+1 ,…,I i+n} represents the coarse-grained data input for continuous frames, where i represents the target time, and i+n and in represent the next n frames and the previous n frames, respectively.

[0073] Take {I} centered on the target frame Ii i-t ,…,I i-t ,I i ,I i+1 ,…,I i+t} A set of coarse-grained data frames, calculate the average value I of this coarse-grained data set. mean The calculation process is as follows:

[0074]

[0075] Where t is the configuration parameter for the external register, t≤n, and the default value is t=1.

[0076] to I i and I mean Perform spatial feature decomposition, I i The calculation process for the high-level spatial information SpaFeature_Hi is as follows:

[0077] SpaFeature_H i {1}(m,n)=(I i (2m-1,2n-1)+I i (2m-1,2n)+I i (2m, 2n-1) + I i (2m,2n)) / 4

[0078] SpaFeature_H i {2}(m,n)=(I i (2m-1,2n-1)-I i (2m-1,2n)+I i (2m,2n-1)-I i (2m,2n) / 4

[0079] SpaFeature_H i {3}(m,n)=(I i (2m-1,2n-1)+I i (2m-1,2n)-I i (2m,2n-1)-I i (2m,2n)) / 4

[0080] SpaFeature_H i {4}(m,n)=(I i (2m-1,2n-1)-I i (2m-1,2n)-I i (2m, 2n-1) + I i (2m,2n)) / 4 (3)

[0082] In this context, () represents the coordinate points of the two values ​​within the parentheses, and {} represents the channel values ​​within the parentheses.

[0083] I mean The calculation process for the high-level spatial information SpaFeature_Hmean is as follows:

[0084] SpaFeature_H mean {1}(m,n)=(I mean(2m-1,2n-1)-I mean (2m-1,2n)+I mean (2m,2n-

[0085] 1)-I mean (2m,2n)) / 4

[0086] SpaFeature_H mean {2}(m,n)=(I mean (2m-1,2n-1)-I mean (2m-1,2n)+I mean (2m,2n-

[0087] 1)-I mean (2m,2n)) / 4

[0088] SpaFeature_H mean {3}(m,n)=(I mean (2m-1,2n-1)+I mean (2m-1,2n)-I mean (2m,2n-

[0089] 1)-I mean (2m,2n)) / 4

[0090] SpaFeature_H mean {4}(m,n)=(I mean (2m-1,2n-1)+I mean (2m-1,2n)-I mean (2m,2n-

[0091] 1)-I mean (2m,2n)) / 4(4)

[0092] In this context, () represents the coordinate points of the two values ​​within the parentheses, and {} represents the channel values ​​within the parentheses.

[0093] I i The calculation process for the low-level spatial information SpaFeature_Li is as follows:

[0094] SpaFeature_L i {1}(m,n)=(SpaFeature_H i {1}(2m-1,2n-1)+SpaFeature_H i {1}(2m-

[0095] 1,2n)+SpaFeature_H i{1}(2m,2n-1)+SpaFeature_H i {1}(2m,2n)) / 4

[0096] SpaFeature_L i {2}(m,n)=(SpaFeature_H i {1}(2m-1,2n-1)+SpaFeature_H i {1}(2m-

[0097] 1,2n)+SpaFeature_H i {1}(2m,2n-1)+SpaFeature_H i {1}(2m,2n)) / 4

[0098] SpaFeature_L i {3}(m,n)=(SpaFeature_H i {1}(2m-1,2n-1)+SpaFeature_H i {1}(2m-

[0099] 1,2n)+SpaFeature_H i {1}(2m,2n-1)+SpaFeature_H i {1}(2m,2n)) / 4

[0100] SpaFeature_L i {4}(m,n)=(SpaFeature_H i {1}(2m-1,2n-1)+SpaFeature_H i {1}(2m-

[0101] 1,2n)+SpaFeature_H i {1}(2m,2n-1)+SpaFeature_H i {1}(2m,2n)) / 4(5)

[0102] In this context, () represents the coordinate points of the two values ​​within the parentheses, and {} represents the channel values ​​within the parentheses.

[0103] I mean Low-level spatial information SpaFeature_L mean The calculation process is as shown in formula (6), and is as follows:

[0104] SpaFeature_L mean {1}(m,n)

[0105] (SpaFeature_L) mean {1}(2m-1,2n-1)+SpaFeature_L mean {1}(2m-1,2n)

[0106] +SpaFeature_L mean {1}(2m,2n-1)+SpaFeature_L mean {1}(2m,2n)) / 4

[0107] SpaFeature_L mean {2}(m,n)

[0108] (SpaFeature_L) mean {1}(2m-1,2n-1)+SpaFeature_L mean {1}(2m-1,2n)

[0109] +SpaFeature_L mean {1}(2m,2n-1)+SpaFeature_L mean {1}(2m,2n)) / 4

[0110] SpaFeature_L mean {3}(m,n)

[0111] (SpaFeature_L) mean {1}(2m-1,2n-1)+SpaFeature_L mean {1}(2m-1,2n)

[0112] +SpaFeature_L mean {1}(2m,2n-1)+SpaFeature_L mean {1}(2m,2n)) / 4

[0113] SpaFeature_L mean {3}(m,n)=(SpaFeature_L mean {1}(2m-1,2n-1)+SpaFeature_L mean {1}(2m-1,2n)+SpaFeature_L mean {1}(2m,2n-1)+SpaFeature_L mean {1}(2m,2n)) / 4 (6)

[0114] In this context, () represents the coordinate points of the two values ​​within the parentheses, and {} represents the channel values ​​within the parentheses.

[0115] Next, S302 is executed, and the variance similarity and matching similarity between the target frame and the average frame in the low-level space are used to approximate the low-level motion features to obtain the motion features of the first-level coarse-grained data in the low-level space.

[0116] In this embodiment, the variance of the target frame low-level spatial data and the average frame low-level spatial data is first calculated. The calculation process is as follows:

[0117]

[0118] In this context, () represents the coordinate points of the two values ​​within the parentheses. The calculation formula is performed using a 7x7 square window centered at (m, n).

[0119] After obtaining the variances of the target frame's low-level spatial data and the average frame's low-level spatial data, variance similarity is calculated. The calculation process is as follows:

[0120] VarSim=1-exp(-α×max(abs(VarI i -VarI mean )-ThVar,0)) (8)

[0121] Here, α and ThVar are externally configured weight mapping parameters used to map variance similarity to the range [0,1]. Specifically, the larger ThVar is, the smaller the VarSim value of regions with more similar variance; the larger α is, the smoother the trend of variance similarity change.

[0122] Then, the low-level spatial data of the target frame and the low-level spatial data of the average frame are matched and similarity is calculated. The calculation process is as follows:

[0123]

[0124] Here, β is an externally configured weight mapping parameter used to map the matching similarity to the range [0, 1]. Specifically, the larger β is, the smoother the trend of matching similarity change.

[0125] Then, step S303 is executed to map the motion representation based on the low-level motion features (i.e., the corresponding variance similarity and matching similarity). The calculation process is as follows:

[0126] MotionRep=VarSim×PatchSim (10)

[0127] Based on the motion representation information, the low-level space of the target frame is iteratively processed to obtain clean low-level space information. The calculation process is as follows:

[0128] SpaFeature_L rep {1}=SpaFeature_L i {1}×MotionRep+SpaFeature_L mean {1}×(1-MotionRep)

[0129] SpaFeature_L rep {2}=SpaFeature_L i {2}×MotionRep+SpaFeature_L mean {2}×(1-MotionRep)

[0130] SpaFeature_L rep {3}=SpaFeature_L i {3}×MotionRep+SpaFeature_L mean {3}×(1-MotionRep)

[0131] SpaFeature_L rep {4}=SpaFeature_L i {4}×MotionRep+SpaFeature_L mean {4}×(1-MotionRep) (11)

[0132] Next, step S304 is executed to construct the low-level residual correction structure. This step aims to preserve the video detail information of the target frame in clean low-level spatial information, further improving the overall video quality. The calculation process is as follows:

[0133] SpaFeature_L rep {i}=

[0134] SpaFeature_L rep {i}+σ×(SpaFeature_L i {i}-SpaFeature_L mean {i})i=0,1,2,3(12)

[0135] Wherein, σ is the residual correction intensity configured externally. Specifically, the larger σ is, the stronger the temporal information of the target frame, but it may cause noise amplification.

[0136] After constructing the low-level residual correction structure, the high-level spatial information SpaFeature_Hrep of the denoised target frame is reconstructed based on the four-channel low-level spatial information. The calculation process is as follows:

[0137] SpaFeature_H rep {1}(2m-1,2n-1)=(SpaFeature_L rep {1}(m,n)+SpaFeature_L rep {2}(m,n)+SpaFeature_L rep {3}(m,n)+SpaFeature_L rep {4}(m,n)

[0138] SpaFeature_H rep {1}(2m-1,2n)=(SpaFeature_L rep {1}(m,n)-SpaFeature_L rep {2}(m,n)+SpaFeature_L rep {3}(m,n)-SpaFeature_L rep {4}(m,n)

[0139] SpaFeature_H rep {1}(2m,2n-1)=(SpaFeature_L rep {1}(m,n)+SpaFeature_L rep {2}(m,n)-SpaFeature_L rep {3}(m,n)-SpaFeature_L rep {4}(m,n)

[0140] SpaFeature_H rep {1}(2m,2n)=(SpaFeature_L rep {1}(m,n)-SpaFeature_L rep {2}(m,n)-SpaFeature_L rep {3}(m,n)+SpaFeature_L rep {4}(m,n)(13)

[0141] Then, S305 is executed, using the variance similarity and matching similarity between the target frame and the average frame in the high-level space of the first-level coarse-grained data to approximate the high-level motion features, thereby obtaining the high-level motion features of the first-level coarse-grained data in the high-level space.

[0142] Specifically, the variance of the target frame high-level spatial data and the average frame high-level spatial data is first calculated. The calculation process is as follows:

[0143]

[0144] In this context, () represents the coordinate points of the two values ​​within the parentheses. The calculation formula is performed using a 7x7 square window centered at (m, n).

[0145] After obtaining the variances of the target frame's high-level spatial data and the average frame's high-level spatial data, variance similarity is calculated. The calculation process is as follows:

[0146] VarSim=1-exp(-α×max(abs(VarI i -VarI mean )-ThVar,0)) (15)

[0147] Here, α and ThVar are externally configured weight mapping parameters used to map variance similarity to the range [0, 1]. Specifically, the larger ThVar is, the smaller the VarSim value of regions with more similar variance; the larger α is, the smoother the trend of variance similarity change.

[0148] Next, the high-level spatial data of the target frame and the high-level spatial data of the average frame are matched and similarity is calculated. The calculation process is as follows:

[0149]

[0150] Here, β is an externally configured weight mapping parameter used to map the matching similarity to the range [0, 1]. Specifically, the larger β is, the smoother the trend of matching similarity change.

[0151] Then, step S306 is executed to map the motion representation based on the high-level motion features (i.e., the corresponding variance similarity and matching similarity). The calculation process is as follows:

[0152] MotionRep=VarSim×PatchSim (17)

[0153] Based on the high-level motion representation information, the high-level space of the target frame is iteratively processed (only channels 2, 3, and 4 are processed here) to obtain clean high-level space information. The calculation process is as follows:

[0154] SpaFeature_H rep {2}=SpaFeature_H i {2}×MotionRep+SpaFeature_H mean{2}×(1-MotionRep)

[0155] SpaFeature_H rep {3}=SpaFeature_H i {3}×MotionRep+SpaFeature_H mean {3}×(1-MotionRep)

[0156] SpaFeature_H rep {4}=SpaFeature_H i {4}×MotionRep+SpaFeature_H mean {4}×(1-MotionRep)(18)

[0157] Next, execute S307 to construct the high-level residual correction structure, as follows:

[0158] SpaFeature_H rep {i}=SpaFeature_H rep {i}+σ×(SpaFeature_H i {i}-SpaFeature_H mean {i})i=0,1,2,3 (18)

[0159] Wherein, σ is the residual correction intensity configured externally. Specifically, the larger σ is, the stronger the temporal information of the target frame, but it may cause noise amplification.

[0160] After constructing the high-level residual correction structure, the first-level coarse-grained data Irep of the denoised target frame is reconstructed based on the four-channel high-level spatial information. The specific calculation is as follows:

[0161] I rep (2m-1,2n-1)=(SpaFeature_H rep {1}(m,n)+SpaFeature_H rep {2}(m,n)+SpaFeature_H rep {3}(m,n)+SpaFeature_H rep {4}(m,n)

[0162] I rep {1}(2m-1,2n)=(SpaFeature_H rep {1}(m,n)-SpaFeature_H rep {2}(m,n)+SpaFeature_H rep{3}(m,n)-SpaFeature_H rep {4}(m,n)

[0163] I rep {1}(2m,2n-1)=(SpaFeature_H rep {1}(m,n)+SpaFeature_H rep {2}(m,n)-SpaFeature_H rep {3}(m,n)-SpaFeature_H rep {4}(m,n)

[0164] I rep {1}(2m,2n)=(SpaFeature_H rep {1}(m,n)-SpaFeature_H rep {2}(m,n)-SpaFeature_H rep {3}(m,n)+SpaFeature_H rep {4}(m,n) (20)

[0165] This invention employs a two-stage coarse-grained denoising framework to denoise the first-level coarse-grained data. It extracts dynamic features from multiple video frames and utilizes the dynamic feature information of adjacent frames to progressively obtain high-quality, clean coarse-grained video frames from lower to higher spatial layers, effectively improving the denoising level of the coarse-grained channel of the video frame. Furthermore, the introduction of a two-stage residual structure during the coarse-grained data denoising process effectively preserves video detail information, further enhancing the overall video quality.

[0166] In this embodiment, a similarity denoising framework is used to denoise the two-level fine-grained data, extract static features from multiple video frames, and obtain high-quality fine-grained clean video frames through local similarity processing.

[0167] For specific details, please refer to... Figure 4 The denoising process for the two-level fine-grained data includes:

[0168] S308. Perform local similarity calculation in the time domain on the secondary fine-grained data;

[0169] S309. Adaptively calculate the denoising intensity based on local similarity information;

[0170] S310. Perform adaptive similarity denoising on the secondary fine-grained level according to the denoising intensity.

[0171] First, we calculate the temporal local similarity of the video sequence based on its fine-grained data input, namely C and K as described in S2. Let {C} i-n ,…,C i-1 C i C i+1 ,…,C i+n Let {K} be the first-level fine-grained data input for consecutive frames, and let {K} be the data input for the first level. i-n ,…,K i-1 ,K i ,K i+1 ,…,K i+n} represents the second-level fine-grained data input for continuous frames, where i represents the target time, and i+n and in represent the next n frames and the previous n frames, respectively.

[0172] We take the target frame C i The central {Ci-t,…,C} i-1 C i C i+1 ,…,C i+t The first-level fine-grained data frame set is used to calculate the average value C of this fine-grained data set. mean The calculation process is as follows:

[0173]

[0174] Where t is the configuration parameter for the external register, t≤n, and the default value is t=1.

[0175] Then take the target frame K i Centered on {Ki-t,…,K i-1 ,K i ,K i+1 ,…,K i+t The second-level fine-grained data frame set is used to calculate the average value K of this fine-grained data set. mean The calculation process is as follows:

[0176]

[0177] Where t is the configuration parameter for the external register, t≤n, and the default value is t=1.

[0178] Based on the fine-grained data of the average frame and the fine-grained data of the target frame, the first-level fine-grained local similarity information LocalSimC1 and LocalSimC2 are calculated, and the second-level fine-grained local similarity information LocalSimK1 and LocalSimK2 are calculated. The calculation process is as follows:

[0179]

[0180]

[0181] In this context, () represents the coordinates of the two values ​​within the parentheses. The calculation formula is performed using a 5x5 square window centered at (m, n).

[0182] Then, S309 is executed to adaptively calculate the denoising intensity based on the local similarity information, as follows;

[0183] DeNoiseStrC=μ×LocalSimC2×exp(-δ×LocalSimC2 2 )

[0184] DeNoiseStrK=μ×LocalSimK2×exp(-δ×LocalSimK2 2 ) (twenty four)

[0185] Where δ is the externally configured compression coefficient, with a default value of 8.173 obtained through experience, and μ is the externally configured denoising strength. Specifically, the larger μ is, the stronger the fine-grained denoising strength of the video.

[0186] Finally, S310 is executed to perform adaptive similarity denoising on the secondary fine-grained data based on the denoising intensity, resulting in clean fine-grained data.

[0187] Specifically, the first-level fine-grained denoising mapping coefficients CoeffC1 and CoeffC2 are calculated first, and the calculation process is as follows:

[0188]

[0189] Based on the denoising mapping coefficients of the first-level fine-grained level, denoising processing is performed on the first-level fine-grained level. The calculation process is as follows:

[0190] C rep =CoeffC1×C mean +CoeffC2 (26)

[0191] Then, the second-level fine-grained denoising mapping coefficients CoeffK1 and CoeffK2 are calculated, as shown in formula (27):

[0192]

[0193] Based on the denoising mapping coefficients of the second-level fine-grained level, denoising processing is performed on the second-level fine-grained level. The calculation process is as follows:

[0194] K rep =CoeffK1×K mean +CoeffK2 (28)

[0195] Finally, the denoised primary coarse-grained information and secondary fine-grained information are reconstructed into correlated luminance and chrominance information through inverse transformation, i.e., reconstructed into YUV video data. The calculation process is as follows:

[0196] Y rep =0.9992×I rep +0.0869×C rep -0.1842×K rep

[0197] U rep = -0.0006×I rep -0.2888×C rep -0.5827×K rep

[0198] V rep = -0.0008×I rep - 0.3860×C rep + 0.5361×K rep (29)

[0199] Among them, Yrep, Urep, and Vrep are the values ​​of the corresponding YUV video output data on the Y, U, and V channels.

[0200] Based on the same inventive concept, please refer to Figure 5 This invention also proposes a video denoising system, comprising:

[0201] The video data input module 100 is configured to input YUV video data of consecutive frames;

[0202] The channel decorrelation transformation module 200 is configured to transform the YUV video data into uncorrelated first-level coarse-grained data and two-level fine-grained data.

[0203] The denoising module 300 is configured to denoise the first-level coarse-grained data and the two-level fine-grained data respectively;

[0204] The channel decorrelation inverse transform module 400 is configured to reconstruct the denoised first-level coarse-grained information and second-level fine-grained information into correlated YUV video data for output through inverse transform.

[0205] Furthermore, the channel decorrelation transformation module 200 applies principal component analysis to approximate the decorrelation transformation matrix, transforming the YUV video data into uncorrelated first-level coarse-grained data and two-level fine-grained data. The calculation formula is as follows:

[0206]

[0207] Wherein, Yin, Uin, and Vin are the matrices corresponding to the YUV video data in the Y, U, and V channels, respectively; I is the coarse-grained matrix obtained after decorrelation transformation; C is the first-level fine-grained matrix obtained after decorrelation transformation; and K is the second-level fine-grained matrix obtained after decorrelation transformation.

[0208] Furthermore, the denoising module 300 includes a coarse-grained denoising unit and a fine-grained denoising unit. The coarse-grained denoising unit is configured to use a two-stage coarse-grained denoising framework to denoise the first-level coarse-grained data, and the fine-grained denoising unit is configured to use a similarity denoising framework to denoise the two-level fine-grained data.

[0209] Furthermore, the coarse-grained denoising unit comprises seven sub-units: a spatial feature decomposition calculation sub-unit 301, a low-level spatial motion feature calculation sub-unit 302, a low-level spatial motion representation mapping sub-unit 303, a low-level spatial residual correction sub-unit 304, a high-level spatial motion feature calculation sub-unit 304, a high-level spatial motion representation mapping sub-unit 306, and a high-level spatial residual correction sub-unit 307. Wherein:

[0210] The spatial feature decomposition calculation subunit 301 is configured to perform spatial feature decomposition on the first-level coarse-grained data to obtain four-channel high-level spatial information and four-channel low-level spatial information.

[0211] The low-level spatial motion feature calculation subunit 302 is configured to approximate the low-level motion features by using the variance similarity and matching similarity between the target frame and the average frame in the low-level space of the first-level coarse-grained data, so as to obtain the motion features of the first-level coarse-grained data in the low-level space.

[0212] The low-level spatial motion representation mapping subunit 303 is configured to map the motion representation according to the low-level motion features to obtain clean low-level spatial information.

[0213] The low-level spatial residual correction subunit 304 is configured to construct a low-level residual correction structure and reconstruct the high-level spatial information of the target frame after denoising based on the four-channel low-level spatial information.

[0214] The high-level spatial motion feature calculation subunit 304 is configured to approximate the high-level motion features by using the variance similarity and matching similarity between the target frame and the average frame in the high-level space of the first-level coarse-grained data, so as to obtain the high-level motion features of the first-level coarse-grained data in the high-level space.

[0215] The high-level space motion representation mapping subunit 306 is configured to map the motion representation according to the high-level motion features to obtain clean high-level space information.

[0216] The high-level spatial residual correction subunit 307 is configured to construct a high-level residual correction structure and reconstruct the first-level coarse-grained data of the target frame after denoising based on the four-channel high-level spatial information.

[0217] This invention employs a two-stage coarse-grained denoising framework to denoise the first-level coarse-grained data. It extracts dynamic features from multiple video frames and utilizes the dynamic feature information of adjacent frames to progressively obtain high-quality, clean coarse-grained video frames from lower to higher spatial layers, effectively improving the denoising level of the coarse-grained channel of the video frame. Furthermore, the introduction of a two-stage residual structure during the coarse-grained data denoising process effectively preserves video detail information, further enhancing the overall video quality.

[0218] In this embodiment, the fine-grained denoising unit uses a similarity denoising framework to denoise the two-level fine-grained data, extracting static features from multiple video frames and obtaining high-quality, clean, fine-grained video frames through local similarity processing. Specifically, the fine-grained denoising unit includes three sub-units: a local similarity calculation sub-unit 308, an adaptive denoising intensity calculation sub-unit 309, and an adaptive similarity denoising sub-unit 310. Wherein:

[0219] The local similarity calculation subunit 308 is configured to perform temporal local similarity calculation on the secondary fine-grained data;

[0220] The adaptive denoising intensity calculation subunit 309 is configured to adaptively calculate the denoising intensity based on local similarity information;

[0221] The adaptive similarity denoising subunit 310 is configured to perform adaptive similarity denoising on the secondary fine-grained level according to the denoising intensity.

[0222] Based on the same inventive concept, embodiments of the present invention also propose a readable storage medium on which a computer program is stored, and when the computer program is executed, it can implement the video denoising method described above.

[0223] A readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction execution device, such as, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer programs described herein can be downloaded from the readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. Networks can include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. Each computing / processing device's network adapter card or network interface receives and forwards a computer program from the network for storage on a readable storage medium within the respective computing / processing device. The computer program used to perform the operations of this invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as "C" or similar languages. The computer program can execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information from a computer program. These electronic circuits can execute computer-readable program instructions, thereby realizing various aspects of the present invention.

[0224] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, systems, and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by a computer program. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. These computer programs can also be stored in a readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the readable storage medium storing the computer program comprises an article of manufacture including instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams.

[0225] A computer program may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the computer program executing on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0226] In summary, the embodiments of the present invention provide a video denoising method, system, and readable storage medium, which have the following beneficial effects:

[0227] 1) By using a decorrelation method, the YUV video data is divided into a first-level coarse-grained channel data and a second-level fine-grained channel data, making each channel independent of each other and avoiding the influence of noise correlation between channels. Then, the first-level coarse-grained data and the second-level fine-grained data are denoised separately, which can effectively improve the denoising level of each channel of the video frame.

[0228] 2) By employing a two-stage coarse-grained denoising framework to denoise the first-level coarse-grained data, dynamic features are extracted from multiple video frames. Utilizing the dynamic feature information of adjacent frames, high-quality, clean coarse-grained video frames are gradually obtained from lower to higher spatial layers, effectively improving the denoising level of the video frame's coarse-grained channel. Furthermore, introducing a two-stage residual structure during the coarse-grained data denoising process effectively preserves video detail information, further enhancing the overall video quality.

[0229] The above description is merely a description of preferred embodiments of the present invention and is not intended to limit the scope of the invention in any way. Any changes or modifications made by those skilled in the art based on the above disclosure are within the protection scope of the present invention. Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the present invention and its equivalents, the present invention also intends to include these modifications and variations.

Claims

1. A video denoising method, characterized in that, include: S100, Input continuous frames of YUV video data; S200: Transform the YUV video data into uncorrelated first-level coarse-grained data and two-level fine-grained data; S300, Denoise the first-level coarse-grained data and the two-level fine-grained data respectively; S400. The denoised first-level coarse-grained information and second-level fine-grained information are reconstructed into correlated YUV video data through inverse transformation and output. Specifically, in step S300, denoising the first-level coarse-grained data includes: S301. Perform spatial feature decomposition on the first-level coarse-grained data to obtain four-channel high-level spatial information and four-channel low-level spatial information. S302. The motion features of the first-level coarse-grained data in the low-level space are approximated by using the variance similarity and matching similarity between the target frame and the average frame in the low-level space. S303. Map the motion representation according to the motion characteristics in the lower space to obtain clean lower space information; S304. Construct a low-level residual correction structure and reconstruct the high-level spatial information of the target frame after denoising based on the four-channel low-level spatial information. S305. The high-level motion features of the first-level coarse-grained data in the high-level space are approximated by using the variance similarity and matching similarity between the target frame and the average frame in the high-level space. S306. Map the motion representation according to the high-level motion characteristics to obtain clean high-level spatial information; S307. Construct a high-level residual correction structure and reconstruct the first-level coarse-grained data of the target frame after denoising based on the four-channel high-level spatial information.

2. The video denoising method according to claim 1, characterized in that, Specifically, S200 includes: The principal component analysis method is used to approximate the decorrelation transformation matrix, and the calculation formula is as follows: Wherein, Yin, Uin, and Vin are the matrices corresponding to the YUV video data in the Y, U, and V channels, respectively; I is the coarse-grained matrix obtained after decorrelation transformation; C is the first-level fine-grained matrix obtained after decorrelation transformation; and K is the second-level fine-grained matrix obtained after decorrelation transformation.

3. The video denoising method according to claim 2, characterized in that, The matrix size corresponding to the Y, U, and V channels of the YUV video data is consistent.

4. The video denoising method according to claim 1, characterized in that, In S300, a two-stage coarse-grained denoising framework is used to denoise the first-level coarse-grained data.

5. The video denoising method according to claim 1, characterized in that, In step S300, a similarity denoising framework is used to denoise the two levels of fine-grained data.

6. The video denoising method according to claim 1 or 5, characterized in that, In step S300, the denoising of the two-level fine-grained data specifically includes: S308. Perform local similarity calculation in the time domain on the secondary fine-grained data; S309. Adaptively calculate the denoising intensity based on local similarity information; S310. Perform adaptive similarity denoising on the secondary fine-grained level according to the denoising intensity.

7. A video denoising system, characterized in that, include: The video data input module is configured to input YUV video data in consecutive frames; The channel decorrelation transformation module is configured to transform the YUV video data into uncorrelated first-level coarse-grained data and two-level fine-grained data. The denoising module is configured to denoise both the first-level coarse-grained data and the two-level fine-grained data. The channel decorrelation inverse transform module is configured to reconstruct the denoised first-level coarse-grained information and second-level fine-grained information into correlated YUV video data for output through inverse transform; Specifically, the denoising module performs denoising on the first-level coarse-grained data by: S301. Perform spatial feature decomposition on the first-level coarse-grained data to obtain four-channel high-level spatial information and four-channel low-level spatial information. S302. The motion features of the first-level coarse-grained data in the low-level space are approximated by using the variance similarity and matching similarity between the target frame and the average frame in the low-level space. S303. Map the motion representation according to the motion characteristics in the lower space to obtain clean lower space information; S304. Construct a low-level residual correction structure and reconstruct the high-level spatial information of the target frame after denoising based on the four-channel low-level spatial information. S305. The high-level motion features of the first-level coarse-grained data in the high-level space are approximated by using the variance similarity and matching similarity between the target frame and the average frame in the high-level space. S306. Map the motion representation according to the high-level motion characteristics to obtain clean high-level spatial information; S307, Constructing a high-level residual correction.

8. The video denoising system according to claim 7, characterized in that, The denoising module includes a coarse-grained denoising unit and a fine-grained denoising unit. The coarse-grained denoising unit is configured to use a two-stage coarse-grained denoising framework to denoise the first-level coarse-grained data, and the fine-grained denoising unit is configured to use a similarity denoising framework to denoise the two-level fine-grained data.

9. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it can implement the video denoising method according to any one of claims 1-6.

Images