Online video denoising system with lookahead loop module
By introducing a leading loop module and alignment module in the video denoising system, denoising using the timing characteristics of the current frame and future frames, the problem that existing systems cannot be processed online is solved, and better video denoising performance and flexibility are achieved.
Patent Information
- Application Number
- CN202111586598.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-21
- Publication Date
- 2025-05-02
- Estimated Expiration
- 2041-12-21
AI Technical Summary
The existing video denoising system based on bidirectional recurrent neural network cannot be processed online and cannot effectively utilize timing information.
An online video denoising system with a leading loop module is designed. The timing characteristics of the current frame and the future frame are extracted through the forward loop module and the leading loop module, and the features of the future frame are aligned to the current frame through the alignment module, and finally the denoising module is fused to the feature.
Effectively utilize historical timing information and adjacent future timing information to realize online video denoising, achieve better video denoising performance, and be able to dynamically adjust the lag time to balance performance and delay according to specific tasks.
Smart Images
Figure CN114331885B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to an online video denoising system with a look-ahead loop module, and belongs to the technical field of video denoising. Background Art
[0002] Video denoising aims to restore clear video from noisy video. The main reason for noise in video is poor recording equipment or low-light environment. With the development and popularity of video applications (such as live broadcast, online conferencing, etc.), obtaining high-quality video content through video denoising has become a research hotspot.
[0003] In recent years, neural network-based image denoising methods have achieved good results, but different network structures still have their own shortcomings. For example, convolutional neural networks cannot use long-distance temporal information; unidirectional recurrent neural networks cannot use future temporal information; bidirectional recurrent neural networks cannot be processed online and cannot be used for online video tasks. Therefore, how to better utilize temporal information and process it online is a problem that needs to be solved urgently. Summary of the invention
[0004] Aiming at the problem that the existing video denoising system based on bidirectional recurrent neural network cannot be processed online, the present invention provides an online video denoising system with a look-ahead recurrent module.
[0005] An online video denoising system with a look-ahead loop module of the present invention comprises:
[0006] A forward loop module is used to process the forward features of the current frame noise image and the adjacent previous frame noise image and the adjacent previous frame noise image to obtain the forward features of the current frame noise image;
[0007] The advance loop module is used to process the advance features of the noise image of the adjacent selected future frame, the previous frame noise image of the adjacent selected future frame, and the previous frame noise image of the adjacent selected future frame to obtain the advance features of the noise image of the adjacent selected future frame;
[0008] An alignment module, used for aligning the advanced features of the future frame noise image from the future frame noise image to the current frame noise image to obtain aligned advanced features;
[0009] The decoding module is used to fuse the forward features of the current frame noisy image with the aligned advanced features to obtain a denoised image of the current frame.
[0010] According to the online video denoising system with a forward loop module of the present invention, the forward loop module comprises:
[0011] The forward optical flow network uses the noise image corresponding to time t as the current frame noise image; the optical flow from time t to time t-1 is obtained from the noise image at time t and the noise image at time t-1;
[0012] The forward reverse deformation unit uses the reverse deformation method to align the forward features at time t-1 to time t according to the optical flow from time t to time t-1, and obtains the forward features from time t-1 to t;
[0013] F forward residual blocks process the current frame noise image and the forward features from time t-1 to t in sequence to obtain the forward features at time t; F is a positive integer.
[0014] According to the online video denoising system with a look-ahead loop module of the present invention, the look-ahead loop module comprises:
[0015] The advanced optical flow network obtains the optical flow from time t+k-1 to time t+k from the noise image at time t+k-1 and the noise image at time t+k; k is the number of the selected future frames;
[0016] The forward deformation unit of the enlarged edge is used to align the advanced feature at time t+k-1 to time t+k using the forward deformation method of the enlarged edge according to the optical flow from time t+k-1 to time t+k, so as to obtain the advanced feature from time t+k-1 to time t+k;
[0017] The edge separation module separates the content and edge of the advanced features from time t+k-1 to time t+k, and obtains the content of the advanced features from time t+k-1 to time t+k and the edge of the advanced features from time t+k-1 to time t+k;
[0018] L advance residual blocks process the noise image at time t+k and the contents of the advance features from time t+k-1 to time t+k in sequence to obtain the advance features at time t+k; L is a positive integer.
[0019] According to the online video denoising system with a look-ahead loop module of the present invention, the alignment module includes k-1 levels of alignment units, and each level of alignment unit includes an alignment reverse deformation unit and an edge merging module;
[0020] In the first-level alignment unit, the edge merging module is used to merge the leading feature at time t+k with the edge of the leading feature from time t+k-1 to time t+k, so as to obtain the first-level merged leading feature; the alignment reverse deformation unit uses the reverse deformation method to align the first-level merged leading feature to time t+k-1 according to the optical flow from time t+k-1 to time t+k, so as to obtain the aligned first-level leading feature;
[0021] In the secondary alignment unit, the edge merging module is used to merge the aligned first-level advanced features with the edges of the advanced features from time t+k-2 to time t+k-1 to obtain the secondary merged advanced features; the alignment reverse deformation unit uses the reverse deformation method to align the secondary merged advanced features to time t+k-2 according to the optical flow from time t+k-2 to time t+k-1 to obtain the aligned second-level advanced features;
[0022] …
[0023] In the k-1 level alignment unit, the edge merging module is used to merge the aligned k-2 level advanced features with the edges of the advanced features from time t to time t+1 to obtain the k-1 level merged advanced features; the alignment reverse deformation unit uses the reverse deformation method to align the k-1 level merged advanced features to time t according to the optical flow from time t to time t+1 to obtain the aligned advanced features from time t+k to time t.
[0024] According to the online video denoising system with a look-ahead loop module of the present invention, the decoding module includes D layers of deep convolution modules, where D is an integer of at least 2; in the first D-1 layers of deep convolution modules, each layer of deep convolution modules is followed by an activation function.
[0025] According to the online video denoising system with a look-ahead loop module of the present invention, the activation function is a ReLU function.
[0026] According to the online video denoising system with a look-ahead loop module of the present invention, at time t, the decoding module sequentially processes the forward features at time t and the look-ahead features aligned from time t+k to time t using a D-layer deep convolution module to obtain a denoised image at time t.
[0027] According to the online video denoising system with a look-ahead loop module of the present invention, the online video denoising system uses a reconstruction loss function Conduct training.
[0028] According to the online video denoising system with the look-ahead loop module of the present invention, the reconstruction loss function for:
[0029]
[0030] In the formula is the denoised image output at time t; x t is the original clear image at time t; T is the number of training video frames.
[0031] According to the online video denoising system with a look-ahead loop module of the present invention, the online video denoising system utilizes a reconstruction loss function The training process is implemented using the Adam optimization algorithm.
[0032] Beneficial effects of the present invention: the forward feature of the present invention contains long-distance historical time series information; the advanced feature contains future time series information of a selected number of future frames; the advanced feature is aligned back to the current frame, and then the forward feature and the aligned advanced feature are fused to obtain the denoising result of the current frame. Therefore, the present invention can effectively utilize the historical time series information and the adjacent future time series information to denoise the noisy video online and achieve better video denoising performance.
[0033] The forward loop module and the look-ahead loop module in the present invention can effectively propagate historical and future timing information, helping to denoise the current frame. At the same time, the processing from the first frame to the current frame (or the adjacent future frame) enables the overall network to perform online video denoising.
[0034] The present invention realizes online video denoising using long-distance historical time series information and near-future time series information for the first time. Compared with mainstream methods, the present invention achieves a better level of quantitative indicators and can perform online denoising. The present invention also has a faster speed and achieves a better compromise between denoising performance and speed. The present invention can dynamically adjust the lag time (i.e., the number of frames using future frames) according to specific tasks to better balance performance and delay.
[0035] The present invention is suitable for online denoising of noisy videos (or video streams). BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 It is a principle block diagram of the online video denoising system with a look-ahead loop module according to the present invention;
[0037] Figure 2 This is the principle block diagram of the forward loop module;
[0038] Figure 3 This is the principle block diagram of the look-ahead loop module;
[0039] Figure 4 This is the principle block diagram of the alignment module;
[0040] Figure 5 It is a schematic diagram of the principle of the edge separation module and the edge merging module;
[0041] Figure 6 It is the principle block diagram of the decoding module. DETAILED DESCRIPTION
[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0043] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.
[0044] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.
[0045] Specific implementation method 1. Combination Figures 1 to 6 As shown, the present invention provides an online video denoising system with a look-ahead loop module, comprising:
[0046] The forward loop module 100 is used to process the forward features of the current frame noise image, the adjacent previous frame noise image and the adjacent previous frame noise image to obtain the forward features of the current frame noise image;
[0047] The advance loop module 200 is used to process the advance features of the noise image of the adjacent selected future frame, the previous frame noise image of the adjacent selected future frame, and the previous frame noise image of the adjacent selected future frame to obtain the advance features of the noise image of the adjacent selected future frame;
[0048] An alignment module 300 is used to align the advanced features of the future frame noise image from the future frame noise image to the current frame noise image to obtain the aligned advanced features;
[0049] The decoding module 400 is used to fuse the forward features of the current frame noisy image with the aligned advanced features to obtain a denoised image of the current frame.
[0050] In this embodiment, the forward loop module 100 processes the noise image from the first frame to the current frame in sequence to obtain the forward feature, which contains historical time sequence information; the forward loop module 200 processes the noise image from the first frame to the adjacent selected number of future frames in sequence to obtain the forward feature, which contains the future time sequence information of the selected number of future frames; the alignment module 300 aligns the forward feature from the adjacent future frame to the current frame; the decoding module 400 merges the forward feature and the aligned forward feature to obtain the denoising result of the current frame. For the forward loop module 100 corresponding to the first frame noise image, its corresponding forward feature can be calculated based on the initial value assigned by experience.
[0051] In this implementation, the forward loop module is used to extract long-distance historical information, and the look-ahead loop module and the alignment module are used to extract near-future information, so that the timing information in the video can be fully utilized to denoise the current frame and obtain a better denoising effect. Both the forward loop module and the look-ahead loop module use an online processing method to process from the first frame to the current frame (or the near future frame), so the entire framework can process noisy videos online.
[0052] Therefore, the video denoising system of the present invention can effectively utilize historical time sequence information and near future time sequence information to achieve better video denoising performance, and can perform video denoising online. The video denoising system of the present invention is suitable for performing online denoising on noisy videos (or video streams).
[0053] Further, combined with Figure 2 As shown, the forward loop module 100 includes:
[0054] At time t, the forward loop module takes the noise image at time t, the noise image at time t-1, and the forward features at time t-1 as input, and outputs the forward features at time t. The details are as follows:
[0055] The forward optical flow network uses the noise image corresponding to time t as the current frame noise image; the optical flow from time t to time t-1 is obtained from the noise image at time t and the noise image at time t-1;
[0056] The forward reverse deformation unit uses the reverse deformation method to align the forward features at time t-1 to time t according to the optical flow from time t to time t-1, and obtains the forward features from time t-1 to t;
[0057] F forward residual blocks process the current frame noise image and the forward features from time t-1 to t in sequence to obtain the forward features at time t; F is a positive integer.
[0058] The forward optical flow network can use the pre-trained PWC-Net and fix the network parameters.
[0059] Going a step further, combined Figure 3 and Figure 5 As shown, the look-ahead loop module 200 includes:
[0060] At time t, the look-ahead loop module takes the noise image at time t+k-1, the noise image at time t+k and the look-ahead feature at time t+k-1 as input to obtain the look-ahead feature at time t+k; then aligns the look-ahead feature at time t+k to time t to obtain the look-ahead feature from time t+k to t. Specifically, it includes:
[0061] The advanced optical flow network obtains the optical flow from time t+k-1 to time t+k from the noise image at time t+k-1 and the noise image at time t+k; k is the number of the selected future frames;
[0062] The forward deformation unit of the enlarged edge is used to align the advanced feature at time t+k-1 to time t+k using the forward deformation method of the enlarged edge according to the optical flow from time t+k-1 to time t+k, and the result of aligning the advanced feature at time t+k-1 to time (t+k) is obtained, that is, the advanced feature from time t+k-1 to time t+k is obtained;
[0063] The edge separation module separates the content and edge of the advanced features from time t+k-1 to time t+k, and obtains the content of the advanced features from time t+k-1 to time t+k and the edge of the advanced features from time t+k-1 to time t+k;
[0064] L advance residual blocks process the noise image at time t+k and the contents of the advance features from time t+k-1 to time t+k in sequence to obtain the advance features at time t+k; L is a positive integer.
[0065] The advanced optical flow network in this embodiment can utilize the forward optical flow network in the forward loop module 100 .
[0066] Going a step further, combined Figure 4 and Figure 5 As shown, the alignment module 300 includes k-1 levels of alignment units, and each level of alignment unit includes an alignment reverse deformation unit and an edge merging module;
[0067] Generally speaking, at time t, the advanced features at time t+k, the edge at time t+k-1, the edge at time t+k-2, ..., the edge at time t+1, the optical flow from time t+k-1 to t+k, the optical flow from time t+k-2 to t+k-1, ..., the optical flow from time t to t+1 are input into the alignment module, and the edge and alignment are restored frame by frame using the reverse deformation and edge merging method. Finally, the advanced features at time t+k are aligned to time t to obtain the advanced features from time t+k to t. Specifically, it includes:
[0068] In the first-level alignment unit, the edge merging module is used to merge the leading feature at time t+k with the edge of the leading feature from time t+k-1 to time t+k, so as to obtain the first-level merged leading feature; the alignment reverse deformation unit uses the reverse deformation method to align the first-level merged leading feature to time t+k-1 according to the optical flow from time t+k-1 to time t+k, so as to obtain the aligned first-level leading feature;
[0069] In the secondary alignment unit, the edge merging module is used to merge the aligned first-level advanced features with the edges of the advanced features from time t+k-2 to time t+k-1 to obtain the secondary merged advanced features; the alignment reverse deformation unit uses the reverse deformation method to align the secondary merged advanced features to time t+k-2 according to the optical flow from time t+k-2 to time t+k-1 to obtain the aligned second-level advanced features;
[0070] …
[0071] In the k-1 level alignment unit, the edge merging module is used to merge the aligned k-2 level advanced features with the edges of the advanced features from time t to time t+1 to obtain the k-1 level merged advanced features; the alignment reverse deformation unit uses the reverse deformation method to align the k-1 level merged advanced features to time t according to the optical flow from time t to time t+1 to obtain the aligned advanced features from time t+k to time t.
[0072] In this embodiment, the edge separation module separates the image or feature into two parts: edge and content; the edge merging module merges the separated edge and content into a complete image or feature.
[0073] The advance features output by the advance loop module of the present invention are aligned with the adjacent future frame, and the alignment module aligns the advance features to the current frame. In the alignment process, there is a problem of missing edge information. The present invention proposes to use forward deformation of enlarged edges for alignment, and save and restore edges during the alignment process, which can better alleviate the problem of missing edges in the alignment process.
[0074] Going a step further, combined Figure 6 As shown, the decoding module 400 includes D layers of depth convolution modules, where D is an integer of at least 2; in the first D-1 layers of depth convolution modules, each layer of depth convolution modules is followed by an activation function, that is, the last layer of depth convolution modules does not have an activation function.
[0075] As an example, the activation function is a ReLU function.
[0076] Going a step further, combined Figure 6 As shown, at time t, the decoding module 400 processes the forward features at time t and the advanced features aligned from time t+k to time t in sequence using a D-layer deep convolution module to obtain a denoised image at time t.
[0077] As a specific embodiment, F=15, L=15, D=2, and k=3 may be selected.
[0078] Furthermore, the online video denoising system of the present invention needs to be trained before use. The online video denoising system uses a reconstruction loss function Conduct training.
[0079] Furthermore, the reconstruction loss function of the denoised image output by the network and the corresponding clear image is Defined as:
[0080]
[0081] In the formula is the denoised image output at time t; x t is the original clear image at time t. During the training process, the noisy image at time t is degraded from the original clear image at time t. The original clear image is obtained from a dataset with paired clear videos and noisy videos. T is the number of training video frames.
[0082] The video denoising system is optimized by image reconstruction loss before use to learn the video denoising system model.
[0083] As an example, an online video denoising system uses the reconstruction loss function The training process is implemented using the Adam optimization algorithm.
[0084] The online video denoising system of the present invention realizes the online video denoising by using long-distance historical time series information and near-future time series information at the same time for the first time. Compared with the mainstream method, the quantitative index of the present invention exceeds the mainstream method, and the speed is also faster, achieving a good compromise between performance and speed. The present invention can dynamically adjust the size of the advance frame number k according to the real-time requirements of the task to achieve a balance between performance and delay.
[0085] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should therefore be understood that many modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in a manner different from that described in the original claims. It should also be understood that the features described in conjunction with a single embodiment may be used in other described embodiments.
Claims
1. An online video denoising system with a look-ahead loop module, characterized in that Includes, A forward loop module (100) is used to process the forward features of the current frame noise image, the adjacent previous frame noise image and the adjacent previous frame noise image to obtain the forward features of the current frame noise image; An advance loop module (200) is used to process the advance features of the adjacent selected future frame noise image, the previous frame noise image of the adjacent selected future frame, and the previous frame noise image of the adjacent selected future frame to obtain the advance features of the adjacent selected future frame noise image; An alignment module (300) is used to align the advanced features of the future frame noise image from the future frame noise image to the current frame noise image to obtain the aligned advanced features; A decoding module (400) is used to fuse the forward features of the current frame noise image and the aligned advanced features to obtain a denoised image of the current frame; The forward loop module (100) comprises: The forward optical flow network uses the noise image corresponding to time t as the current frame noise image; the optical flow from time t to time t-1 is obtained from the noise image at time t and the noise image at time t-1; The forward reverse deformation unit uses the reverse deformation method to align the forward features at time t-1 to time t according to the optical flow from time t to time t-1, and obtains the forward features from time t-1 to t; F forward residual blocks process the current frame noise image and the forward features from time t-1 to t in sequence to obtain the forward features at time t; F is a positive integer.
2. The online video denoising system with a look-ahead loop module according to claim 1, characterized in that: The look-ahead loop module (200) comprises: The advanced optical flow network obtains the optical flow from time t+k-1 to time t+k from the noise image at time t+k-1 and the noise image at time t+k; k is the number of the selected future frames; The forward deformation unit of the enlarged edge is used to align the advanced feature at time t+k-1 to time t+k using the forward deformation method of the enlarged edge according to the optical flow from time t+k-1 to time t+k, so as to obtain the advanced feature from time t+k-1 to time t+k; The edge separation module separates the content and edge of the advanced features from time t+k-1 to time t+k, and obtains the content of the advanced features from time t+k-1 to time t+k and the edge of the advanced features from time t+k-1 to time t+k; L advance residual blocks process the noise image at time t+k and the contents of the advance features from time t+k-1 to time t+k in sequence to obtain the advance features at time t+k; L is a positive integer.
3. The online video denoising system with a look-ahead loop module according to claim 2, characterized in that: The alignment module (300) comprises k-1 levels of alignment units, each level of alignment units comprising an alignment reverse deformation unit and an edge merging module; In the first-level alignment unit, the edge merging module is used to merge the leading feature at time t+k with the edge of the leading feature from time t+k-1 to time t+k to obtain the first-level merged leading feature; The alignment reverse deformation unit uses the reverse deformation method to align the advanced features after the first-level merging to the time t+k-1 according to the optical flow from time t+k-1 to time t+k, and obtains the aligned first-level advanced features; In the secondary alignment unit, the edge merging module is used to merge the aligned primary leading feature with the edge of the leading feature from time t+k-2 to time t+k-1 to obtain the secondary merged leading feature; The alignment reverse deformation unit uses the reverse deformation method to align the advanced features after the secondary merger to the time t+k-2 according to the optical flow from time t+k-2 to time t+k-1, and obtains the aligned secondary advanced features; …… In the k-1 level alignment unit, the edge merging module is used to merge the aligned k-2 level advanced features with the edge of the advanced features from time t to time t+1 to obtain the k-1 level merged advanced features; The alignment reverse deformation unit uses the reverse deformation method to align the k-1 level merged advanced features to time t according to the optical flow from time t to time t+1, and obtains the advanced features aligned from time t+k to time t.
4. The online video denoising system with a look-ahead loop module according to claim 3, characterized in that: The decoding module (400) comprises D layers of depth convolution modules, where D is an integer of at least 2; in the first D-1 layers of depth convolution modules, each layer of depth convolution modules is followed by an activation function.
5. The online video denoising system with a look-ahead loop module according to claim 4, characterized in that: The activation function is the ReLU function.
6. The online video denoising system with a look-ahead loop module according to claim 5, characterized in that: At time t, the decoding module (400) sequentially processes the forward features at time t and the advanced features aligned from time t+k to time t using a D-layer deep convolution module to obtain a denoised image at time t.
7. The online video denoising system with a look-ahead loop module according to any one of claims 1 to 6, characterized in that: The online video denoising system uses a reconstruction loss function Conduct training.
8. The online video denoising system with a look-ahead loop module according to claim 7, characterized in that: Reconstruction loss function for: In the formula is the denoised image output at time t; x t is the original clear image at time t; T is the number of training video frames.
9. The online video denoising system with a look-ahead loop module according to claim 7, characterized in that: Online video denoising system using reconstruction loss function The training process is implemented using the Adam optimization algorithm.
Citation Information
Patent Citations
Video blind denoising method and device based on deep learning
CN111539879A