Video noise reduction method and device, computer system and readable storage medium

Abstract

The present invention provides a video denoising method and device, a computer system and a readable storage medium. The denoising method comprises: obtaining a video stream, wherein the video stream comprises a plurality of image frames arranged in time sequence; generating an optical flow field between adjacent image frames; based on the optical flow field, constructing a foreground image and performing video decomposition to generate a foreground and background corresponding to the image frame; performing denoising processing on the foreground and background respectively to obtain the denoised foreground and background, and generating a denoised foreground; generating a denoised video stream based on the denoised foreground and background; the neural network system can perform denoising processing on the video stream.

Description

Technical Field

[0001] The present invention relates to the technical field of video noise reduction, and in particular to a video noise reduction method and device, a computer system and a readable storage medium. Background Art

[0002] In modern society, it is common to use cameras or detectors to shoot videos. When shooting videos, due to the influence of various factors, the obtained video streams often contain noise, such as Gaussian noise, compression noise, etc. In the long-term practice of the inventor, it is found that when shooting videos, cameras or detectors are usually not stationary, but are in vibration or motion, which causes the background in the captured video to move, which will cause noise. Therefore, it is necessary to perform noise reduction processing on the videos captured in this situation.

[0003] In the existing technology, the following technologies are usually used for video noise reduction:

[0004] (1) Denoising methods based on supervised learning. Most of these denoising methods perform denoising on each image frame of the video stream. That is, these denoising methods do not make good use of the information between adjacent image frames and the temporal consistency, resulting in poor denoising effect.

[0005] (2) RPCA (Robust Principle Component Analysis) decomposition method: This method can only model the foreground and background of videos with static backgrounds, but cannot process videos with moving backgrounds caused by camera lens motion. Summary of the invention

[0006] The object of the present invention is to provide a video noise reduction method and device, a computer system and a readable storage medium.

[0007] In order to achieve one of the above-mentioned objects, an embodiment of the present invention provides a video noise reduction method, comprising the following steps: obtaining a video stream, wherein the video stream includes a plurality of image frames I0, I1, ..., I2 arranged in time sequence. T , T is a natural number; generate image frame I m and image frame I m-1 The optical flow field O m→m-1 =G flow (I m , I m-1 ), where G flow is the optical flow estimator; based on the optical flow field O m→m-1 , construct the foreground image and perform video decomposition, and generate image frame I m The corresponding foreground S m and background Lm ; Foreground S m and background L m Perform noise reduction processing separately and obtain the noise-reduced foreground And the background after noise reduction The denoised foreground is generated based on the following formula Where m = 1, 2, ..., T, W() is the registration function, ρ is the smoothness value; based on the foreground and background Generate a denoised video stream, where m=1, 2, ..., T.

[0008] As a further improvement of the embodiment of the present invention, the “based on optical flow field O m→m-1 , construct the foreground image and perform video decomposition, and generate image frame I m The corresponding foreground S m and background L m "Includes: Generate I m with I m-1 Based on the background offset, generate the current image frame I m Relative to the previous image frame I m-1 The homography matrix H between m→m-1 ; Generate image frame I using homography matrix m The corresponding panoramic size image Y m And the mask matrix Based on panoramic image, principal component tracking algorithm and optical flow field O m→m-1 , generate image frame I m The corresponding foreground S m and background L m .

[0009] As a further improvement of the embodiment of the present invention, the mask matrix For image frame I m Obtained by performing a perspective transformation.

[0010] As a further improvement of the embodiment of the present invention, the “generating m with I m-1 The background offset between the two includes: generating a kernel density estimator based on the optical flow field P m→m-1 The probability density function of the window with the largest probability density value is selected as I m with I m-1 The background offset between .

[0011] As a further improvement of the embodiment of the present invention, the “foreground S m and background L mPerform noise reduction processing separately and obtain the noise-reduced foreground And the background after noise reduction "Includes: Based on single-frame self-supervised denoising network Foreground S m and background L m Perform noise reduction processing separately and obtain the noise-reduced foreground And the background after noise reduction

[0012] As a further improvement of the embodiment of the present invention, the “foreground S m and background L m Perform noise reduction processing separately and obtain the noise-reduced foreground And the background after noise reduction "Includes: Based on the self2self network, foreground S m and background L m Perform noise reduction processing separately and obtain the noise-reduced foreground And the background after noise reduction

[0013] As a further improvement of the embodiment of the present invention, the “based on optical flow field O m→m-1 , construct the foreground image and perform video decomposition, and generate image frame I m The corresponding foreground S m and background L m "Includes: Based on principal component tracking algorithm and optical flow field O m→m-1 , generate image frame I m The corresponding foreground S m and background L m .

[0014] The embodiment of the present invention further provides a video noise reduction device, comprising the following modules: a data acquisition module, for acquiring a video stream, wherein the video stream includes a plurality of image frames I0, I1, ..., I2 arranged in time sequence. T , T is a natural number; data processing module, used to generate image frame I m and image frame I m-1 The optical flow field O m→m-1 =G flow (I m , I m-1 ), where G flow is the optical flow estimator; based on the optical flow field O m→m-1 , construct the foreground image and perform video decomposition, and generate image frame I m The corresponding foreground S m and background L m ; Foreground S m and background L mPerform noise reduction processing separately and obtain the noise-reduced foreground And the background after noise reduction The denoised foreground is generated based on the following formula Where m = 1, 2, ..., T, W() is the registration function, ρ is the smoothness value; the video stream generation module is used to generate the foreground and background Generate a denoised video stream, where m=1, 2, ..., T.

[0015] An embodiment of the present invention also provides a computer system, comprising: one or more processors; a computer-readable storage medium for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned method.

[0016] An embodiment of the present invention further provides a readable storage medium on which executable instructions are stored. When the instructions are executed by a processor, the processor implements the above method.

[0017] Compared with the prior art, the technical effect of the present invention is that: the embodiments of the present invention provide a video denoising method and device, a computer system and a readable storage medium, the denoising method comprising: obtaining a video stream, the video stream comprising a plurality of image frames arranged in time sequence; generating an optical flow field between adjacent image frames; based on the optical flow field, constructing a foreground map and performing video decomposition to generate a foreground and background corresponding to the image frame; performing denoising processing on the foreground and background respectively and obtaining the denoised foreground and background to generate a denoised foreground; generating a denoised video stream based on the denoised foreground and background; the neural network system can perform denoising processing on the video stream. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flow chart of a video noise reduction method in an embodiment of the present invention;

[0019] Figure 2 is a principle diagram of a video noise reduction method in an embodiment of the present invention;

[0020] Figure 3 and Figure 4 3 is a diagram showing the experimental results of the video noise reduction method in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The present invention will be described in detail below in conjunction with the various embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional changes made by a person skilled in the art based on these embodiments are all within the scope of protection of the present invention.

[0022] As used herein, terms indicating spatial relative positions such as "upper," "above," "lower," and "below" are used for the purpose of convenience to describe the relationship of one unit or feature relative to another unit or feature as shown in the accompanying drawings. Terms of spatial relative position may be intended to include different orientations of the device in use or operation in addition to the orientation shown in the drawings. For example, if the device in the figure is turned over, units described as being "below" or "beneath" other units or features will be located "above" the other units or features. Thus, the exemplary term "below" can encompass both above and below orientations. The device may be oriented in other ways (rotated 90 degrees or in other orientations), and the spatially relative descriptors used herein interpreted accordingly.

[0023] Furthermore, it should be understood that although the terms first, second, etc. may be used herein to describe various elements or structures, these described objects should not be limited by these terms. These terms are only used to distinguish these described objects from each other. For example, a first data acquisition module may be referred to as a second data acquisition module, and similarly, a second data acquisition module may also be referred to as a first data acquisition module, without departing from the scope of protection of this application.

[0024] Embodiment 1 of the present invention provides a video noise reduction method, such as Figure 1 As shown, the following steps are included:

[0025] Step 101: Obtain a video stream, wherein the video stream includes a plurality of image frames I0, I1, ..., I2 arranged in time sequence. T , T is a natural number; here, it can be understood that the video stream is composed of a number of image frames, which are arranged in chronological order. The video stream {I0, I1, ..., I T}∈R M×N×T , where M and N represent the length and width of the image frame respectively, and R is a set of real numbers.

[0026] It is understandable that when shooting a video stream, if the target object moves, the user usually turns the lens to track the target object, which causes the background to move, thereby destroying the temporal consistency of the video stream. Here, in order to maintain temporal consistency, each image frame in the video stream needs to construct a corresponding panoramic image.

[0027] To facilitate calculation and understanding, we can first construct a world view canvas and introduce the first image frame I0 as the reference frame of the entire video. That is, the background of the first image frame I0 is used as the world view canvas. Then the world coordinates of the first image frame I0 are [0, 0].

[0028] Step 102: Generate image frame Im and image frame I m-1 The optical flow field O m→m-1 =G flow (I m , I m-1 ), where G flow is the optical flow estimator; based on the optical flow field O m→m-1 , construct the foreground image and perform video decomposition, and generate image frame I m The corresponding foreground S m and background L m ; Foreground S m and background L m Perform noise reduction processing separately and obtain the noise-reduced foreground And the background after noise reduction The denoised foreground is generated based on the following formula Where m = 1, 2, ..., T, W() is the registration function, and ρ is the smoothness value;

[0029] Here, a single-frame self-supervised denoising network can be used To reduce the noise of the foreground m With background L m , and the noise reduction results are recorded as and You can also use the self2self self-supervised denoising network to achieve single-frame denoising.

[0030] Here, ρ can be used to control the smoothness. A larger ρ will produce a smoother denoising result.

[0031] Step 103: Based on the foreground and background Generate a denoised video stream, where m=1, 2, ..., T.

[0032] In this embodiment, the “based on optical flow field O m→m-1 , construct the foreground image and perform video decomposition, and generate image frame I m The corresponding foreground S m and background L m "include:

[0033] Generate I m with I m-1 Based on the background offset, generate the current image frame I m Relative to the previous image frame I m-1 The homography matrix H between m→m-1 ; Generate image frame I using homography matrix m The corresponding panoramic image size Y m And the mask matrix Based on panoramic image, principal component tracking algorithm and optical flow field O m→m-1 , generate image frame I m The corresponding foreground S m and background L m .

[0034] Based on the background offset, the current image frame I is generated m Relative to the previous image frame I m-1 The homography matrix H between m→m-1 Similarly, the current image frame I can also be calculated m The homography matrix H between the first image frame I0 m→0 In addition, you can also get a panoramic current picture Y m and an I obtained by perspective transformation m The mask matrix It can be understood that based on the homography matrix, each image is placed at a specific fixed position in the world view canvas, which preserves the temporal consistency of the video.

[0035] Here, we first explain the effectiveness of video decomposition for video noise reduction: assuming that the image frame Y contains noise, and Y can be decomposed into Y=L+S, where L is the background and S is the foreground. It can be understood that L and S can be understood as an M*N matrix, where M and N are the length and height of Y respectively.

[0036] Given a As a mask matrix marking the non-zero elements in the S matrix, according to the sparsity of the foreground, there is The background L is generated based on the low-rank assumption, which will not change significantly with the change of the time axis, which means that most of the noise energy will be concentrated on the foreground while the background noise energy is relatively low. Therefore, if the low-noise background L is denoised, Denoise the foreground S to get You can define So The noise energy of will definitely not exceed the noise energy of directly denoising the polluted image frame Y, denoted as Assuming X is the clean data of the noisy image Y, it can be proved that Among them, ⊙ is the element-by-element multiplication.

[0037] Here, the background offset will cause the two consecutive image frames to not be in the same position on the world view canvas, introducing the symbol and Represent non-overlapping areas and overlapping areas respectively. However, in general, non-overlapping areas generally introduce new information, which makes and Therefore, there is

[0038]

[0039] Among them, since the prospect is zero, is also all zero, eliminating it from the formula for the non-overlapping region. and The errors of the denoising results are recorded as as well as So there is This means that the result of video decomposition is conducive to noise reduction.

[0040] In this embodiment, the mask matrix For image frame I m Obtained by performing a perspective transformation.

[0041] In this embodiment, the “generating m with I m-1 The background offset between the two includes: generating a kernel density estimator based on the optical flow field O m→m-1 The probability density function of the window with the largest probability density value is selected as I m with I m-1 The background offset between .

[0042] Here, it can be understood that the optical flow field P m→m-1 ∈R M×N×2 It can be understood that although the number of pixels of the moving target object is greater than that of the background, the moving direction of the optical flow of the target object's pixels is scattered and chaotic, while the moving direction of the optical flow of the background is highly consistent. Therefore, the kernel density estimator can be used to construct a probability density function, which can then describe the optical flow field P. m→m-1 , select the value with the largest probability density [u max , v max ] corresponding to the window displacement [x, y] as the current image frame I m Relative to the previous image frame I m-1 The background offset between .

[0043] In this embodiment, the “foreground S m and background L m Perform noise reduction processing separately and obtain the noise-reduced foreground And the background after noise reduction "include:

[0044] Single-frame self-supervised denoising network Foreground S m and background L mPerform noise reduction processing separately and obtain the noise-reduced foreground And the background after noise reduction

[0045] In this embodiment, the “foreground S m and background L m Perform noise reduction processing separately and obtain the noise-reduced foreground And the background after noise reduction "include:

[0046] Based on the self2self network, the foreground S m and background L m Perform noise reduction processing separately and obtain the noise-reduced foreground And the background after noise reduction

[0047] In the training phase of the self2self network, we can first use Bernoulli sampling to generate many training samples, denoted as in So the definition And b is the mask matrix of downsampling, so by minimizing the following loss function To train this network:

[0048] In the testing phase, in order to reduce the time spent on prediction, a U-Net is used as the student network The denoising ability of self2self is learned through fully supervised training, and its loss function is

[0049] Finally, a spatial-temporal double-sideband filter based on optical flow is used to achieve multi-frame denoising. Optical flow field Depend on Therefore, a bilateral filter is used to average the 2K+1 registered image frames to obtain the denoised result.

[0050] In this embodiment, the “based on the optical flow field P m→m-1 , construct the foreground image and perform video decomposition, and generate image frame I m The corresponding foreground S m and background L m "Includes: Based on principal component tracking algorithm and optical flow field O m→m-1 , generate image frame I m The corresponding foreground S m and background L m .

[0051] Based on the theory mentioned above, in this stage, the panoramic frame Y∈R constructed in the previous stage will be MN×1 Decompose into a low-rank background L∈R MN×1 With sparse foreground S∈R MN×1 .

[0052] In the principal component tracking algorithm, there are four operations, namely PartialSVD, IncSVD, RepSVD and DwnSVD. These operations are used to maintain and update the ∑ matrix, U matrix and V matrix in this algorithm, and finally the background matrix is ​​calculated by ∑UV(-1,:). However, before decomposition, since two consecutive panoramic frames are not in the same place in the world view, the following two formulas are needed to fill the non-overlapping area:

[0053]

[0054] in represents the unknown area in U relative to Y, and represents the unknown area in Y relative to U. ()+ represents the pseudo-inverse matrix of a matrix. The upper part of the formula is used to complete the non-overlapping area in Y, while the lower part of the formula is used to complete the non-overlapping area in U. After the matrix U and the matrix Y complement each other, the principal component pursuit algorithm mentioned above is used to decompose Y.

[0055] The principal component tracking algorithm can decompose the video in real time in a frame-by-frame manner. The component tracking algorithm solves the following relaxations: This problem can be solved by the principal component tracking algorithm, which finally returns a low-rank matrix B and a sparse matrix F, which are further reshaped into the background B and foreground F of the video.

[0056] In the principal component tracking algorithm, partialSVD, incSVD, repSVD and downSDV are variants of singular value decomposition (SVD). The entire algorithm flow of the principal component tracking algorithm is shown in the following table:

[0057] surface

[0058]

[0059] To sum up, video decomposition has advantages for noise reduction. Video decomposition can bring two major advantages: 1. After decomposition, the video is decomposed into foreground and background, and the noise on the image frame is correspondingly divided into two parts: background noise and foreground noise. Since the background has always been a low-rank matrix in time, it can be considered that the energy of the noise related to the background is very low, otherwise the low-rank assumption will no longer hold. This will make the background have a very high signal-to-noise ratio, making the noise reduction task simpler, which is also the conclusion found by the inventor in the experiment; 2. The foreground retains most of the noise energy in the original image. Since the foreground contains several sparse moving objects, the foreground noise is largely separated from the image content, making the foreground noise easier to remove.

[0060] In order to test the noise reduction performance of the neural network system, the inventor built a blood vessel angiography video data set with true clean data for testing. This data set has a total of 27 video data (1568 image frames), including 20 static video data and 7 motion video data. High-dose X-Ray angiography data was selected as the real data, and Gaussian noise was artificially added to simulate the pollution pictures in the real environment.

[0061] In the training phase, self2self is used as a denoiser and the Bernoulli sampling probability is set to 0.3. Then a student network is trained using the Adam optimizer with a learning rate of 10-4 for 100 epochs, with 8 samples per training. In the Stabilize phase, the world view canvas size is 2048×2048 and the image frame size M×N is 1024×1024. In the Inc-PCP algorithm, it is implemented on the GPU except for the code to fill in the blanks and set rank=1. The parameters And the window size is 30 frames. In the spatiotemporal double sideband filter, the parameter ρ is set to 0.02. All experiments will be completed on NVIDIA GeForce RTX2080Ti GPU.

[0062] Self-supervised deep models were selected as benchmarks, including Noise2Void, Noise2Self, and Self2Self. Since this dataset includes clean true value data, PSNR and SSIM can be used as objective quantitative indicators to reflect the performance of each model. Table 1 shows the performance comparison between the various models. The PSNR and SSIM scores of the present invention are the best among these self-supervised learning models, with an improvement of 5-6 dB, especially compared with Self2Self, which shows the effectiveness of the Stabilize stage and the Decompose stage. Similarly, according to SSIM, the model is also the best. It is worth noting that as the noise energy increases, the present invention can always maintain the best score and has good robustness. Figure 3 It is the result of visual processing.

[0063] The following table compares the denoising results of the self-supervised denoising method. The italicized text indicates that the neural network system used different denoising methods in the denoising stage. The bolded part shows the best score.

[0064]

[0065] The following table compares the three-stage noise reduction based on S2S and the framework without stabilization stage

[0066]

[0067] First, the motion of the camera lens will cause the background of the video to shift, which will cause the background obtained after the video decomposition to be severely blurred. In order to quantify this phenomenon, the image entropy of each background is calculated as a method to measure the stabilization stage to verify the effectiveness of this stage in preserving the temporal consistency of the video. The inventors plotted the image entropy curve and compared the results with and without the stabilization stage. Figure 4 As shown in Table 1. This clearly shows the information retention of the background image in the Stabilize stage. Table 2 also confirms this phenomenon. In addition, in order to verify the generalization performance of the present invention, the denoiser in the present invention is replaced by Noise2Void and Noise2Self. Their results are shown in Table 1, which clearly proves the effectiveness of the neural network system.

[0068] Embodiment 2 of the present invention provides a video noise reduction device, comprising the following modules:

[0069] The data acquisition module is used to acquire a video stream, wherein the video stream includes a plurality of image frames I0, I1, ..., I2 arranged in time sequence. T , T is a natural number;

[0070] Data processing module, used to generate image frame Im and image frame I m-1 The optical flow field O m→m-1 =G flow (I m , I m-1 ), where G flow is the optical flow estimator; based on the optical flow field O m→m-1 , construct the foreground image and perform video decomposition, and generate image frame I m The corresponding foreground S m and background L m ; Foreground S m and background L m Perform noise reduction processing separately and obtain the noise-reduced foreground And the background after noise reduction The denoised foreground is generated based on the following formula Where m = 1, 2, ..., T, W() is the registration function, and ρ is the smoothness value;

[0071] Video stream generation module for foreground-based and background Generate a denoised video stream, where m=1, 2, ..., T.

[0072] Embodiment 3 of the present invention provides a computer system, including:

[0073] one or more processors;

[0074] A computer readable storage medium for storing one or more programs,

[0075] When the one or more programs are executed by the one or more processors, the one or more processors implement the method in embodiment 1.

[0076] Embodiment 4 of the present invention provides a readable storage medium on which executable instructions are stored. When the instructions are executed by a processor, the processor implements the method in embodiment 1.

[0077] It should be understood that although this specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each implementation mode may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

[0078] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. Any equivalent implementation methods or changes that do not deviate from the technical spirit of the present invention should be included in the scope of protection of the present invention.

Claims

1. A video noise reduction method, It is characterized in that The following steps are involved: Acquire a video stream, wherein the video stream contains multiple image frames arranged in time sequence , , …, , T is a natural number; Generate image frame and image frames The optical flow field between ,in, is an optical flow estimator; based on the kernel density estimator, an optical flow field is generated to describe the optical flow field The probability density function of and Based on the background offset, generate the current image frame Relative to the previous image frame The homography matrix between ; Generate image frames using homography matrix The corresponding panoramic size image And the mask matrix ; Based on panorama, principal component tracking algorithm and optical flow field , generate image frames The corresponding prospects and background , the mask matrix For image frames Obtained by perspective transformation; Based on a single-frame self-supervised denoising network or self2self network, for prospect and background Perform noise reduction processing separately and obtain the noise-reduced foreground And the background after noise reduction , the denoised foreground is generated based on the following formula , , where m=1,2,…,T, is the registration function, is the smoothness value; Based on the prospect and background Generate a denoised video stream, where m=1,2,…,T.

2. A video noise reduction device, It is characterized in that Includes the following modules: A data acquisition module is used to acquire a video stream, wherein the video stream contains multiple image frames arranged in time sequence. , , …, , T is a natural number; Data processing module, used to generate image frames and image frames The optical flow field between ,in, is an optical flow estimator; based on the kernel density estimator, an optical flow field is generated to describe the optical flow field The probability density function of and Based on the background offset, generate the current image frame Relative to the previous image frame The homography matrix between ; Generate image frames using homography matrix The corresponding panoramic size image And the mask matrix ; Based on panorama, principal component tracking algorithm and optical flow field , generate image frames The corresponding prospects and background The mask matrix For image frames Obtained by perspective transformation; Based on a single-frame self-supervised denoising network or self2self network, for prospect and background Perform noise reduction processing separately and obtain the noise-reduced foreground And the background after noise reduction , the denoised foreground is generated based on the following formula , , where m=1,2,…,T, is the registration function, is the smoothness value; Video stream generation module for foreground-based and background Generate a denoised video stream, where m=1,2,…,T.

3. A computer system, It is characterized in that include: one or more processors; A computer-readable storage medium for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the video noise reduction method of claim 1.

4. A readable storage medium, It is characterized in that Executable instructions are stored thereon, and when the instructions are executed by a processor, the processor implements the video noise reduction method in claim 1.

Images