All-in-one video restoration method and system based on reliability gating mechanism guidance

Abstract

The application discloses an ALL-in-One video restoration method and system based on a reliability gating mechanism guide, a motion reliability gating module is introduced to explicitly model the reliability of the propagation feature, effectively suppressing invalid information propagation caused by complex motion and false alignment; meanwhile, a frequency domain detail refinement module is combined to enhance the edge and texture recovery capability, so that clear, stable and good real scene adaptability restoration results are realized in various video degradation tasks.

Description

Technical Field

[0001] This invention belongs to the field of video processing technology, specifically relating to an ALL-in-One video restoration method and system guided by a reliability gating mechanism. Background Technology

[0002] With the rapid development of applications such as mobile terminals, intelligent monitoring, network video platforms, and autonomous driving, video data has become an important information carrier. However, during actual acquisition, transmission, and storage, videos are often affected by various degradation factors, such as low-resolution sampling, motion blur, sensor noise, compression artifacts, low-light imaging, and inter-frame inconsistencies caused by complex motion. These degradations significantly reduce the visual quality of videos and further affect the performance of subsequent high-level vision tasks such as object detection, recognition, and tracking. Therefore, how to recover clear, stable, and detailed high-quality videos from low-quality video sequences is an important research problem in the fields of computer vision and image processing. Traditional video restoration methods usually model a single type of degradation, such as methods specifically for video denoising or video deblurring. These methods can achieve certain results in specific tasks, but when faced with situations where multiple degradations coexist in real-world scenarios, it is often necessary to train or deploy multiple models separately, resulting in insufficient generalization ability, high usage costs, and poor flexibility in practical applications. In recent years, all-in-one video restoration methods have received increasing attention. Their goal is to improve the applicability of the model in real-world scenarios by using a unified model to handle various video degradation tasks.

[0003] However, existing all-in-one video restoration methods still face the following core challenges: First, complex motion, occlusion, and optical flow estimation errors lead to inaccurate inter-frame alignment, which can easily introduce unreliable historical information, resulting in ghosting, edge misalignment, and texture contamination. Second, different degradation tasks have significantly different feature requirements, and the lack of explicit constraints on the reliability of propagated features can easily lead to feature misuse or incorrect propagation. Third, insufficient modeling of frequency domain details results in overly smooth restoration results with unclear high-frequency textures. Fourth, some high-performance models have a large number of parameters and computational cost, making them difficult to deploy in practice. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention provides an ALL-in-One video restoration method and system guided by a reliability gating mechanism. By improving the feature propagation mechanism and detail enhancement method, the efficiency of the model in utilizing temporal features and the ability to recover high-frequency details in complex degradation scenarios are improved, thereby realizing ALL-in-One video restoration based on a reliability gating mechanism that is adaptable to various degradation tasks.

[0005] To achieve the above objectives, the present invention provides the following solution: An all-in-one video restoration method guided by a reliability gating mechanism includes: Step S1: Acquire continuous low-quality video data; Step S2: Input continuous low-quality video data into the video restoration model, model the credibility of temporal propagation features through the motion reliability gating module, and enhance the edge and texture details of the restored video frames by combining the frequency domain / high-frequency detail refinement module, so as to achieve unified processing of multiple video restoration tasks such as video denoising, deblurring and compression artifact removal.

[0006] As a preferred approach, the video restoration model is based on a cyclic propagation video restoration network, and introduces a motion reliability gating module (MRG) and a frequency domain detail refinement module (FDR) to achieve temporal alignment stability and detail recovery capability in various video degradation tasks.

[0007] Preferably, the video restoration model includes: a DeformAlignment module, a Motion Reliability Gating Module (MRG), and a Frequency Domain Detail Refinement Module (FDR); wherein, the DeformAlignment module is used to spatially align the features of adjacent frames during forward and backward propagation; the MRG module is used to estimate the reliability of propagation features using a lightweight gating structure, generating a spatially relevant reliability weight map based on the current frame features, propagation features, and motion information; and the FDR module is used to enhance high-frequency details of edges and textures during the image reconstruction stage.

[0008] This invention also provides an ALL-in-One video restoration system guided by a reliability gating mechanism, comprising: The first processing module is used to acquire continuous low-quality video data; The second processing module is used to input continuous low-quality video data into the video restoration model, model the credibility of temporal propagation features through the motion reliability gating module, and enhance the edge and texture details of the restored video frames by combining the frequency domain / high-frequency detail refinement module, so as to achieve unified processing of multiple video restoration tasks such as video denoising, deblurring and compression artifact removal.

[0009] As a preferred approach, the video restoration model is based on a cyclic propagation video restoration network, and introduces a motion reliability gating module (MRG) and a frequency domain detail refinement module (FDR) to achieve temporal alignment stability and detail recovery capability in various video degradation tasks.

[0010] Preferably, the video restoration model includes: a DeformAlignment module, a Motion Reliability Gating Module (MRG), and a Frequency Domain Detail Refinement Module (FDR); wherein, the DeformAlignment module is used to spatially align the features of adjacent frames during forward and backward propagation; the MRG module is used to estimate the reliability of propagation features using a lightweight gating structure, generating a spatially relevant reliability weight map based on the current frame features, propagation features, and motion information; and the FDR module is used to enhance high-frequency details of edges and textures during the image reconstruction stage.

[0011] Compared with the prior art, the beneficial effects of the present invention are as follows: First, by introducing a motion reliability gating module, the reliability of temporal propagation features can be adaptively judged. This can effectively reduce the impact of unreliable historical features on the current frame restoration result when there are complex motions, occlusions, and inter-frame alignment errors, thereby reducing problems such as ghosting, edge misalignment, and texture contamination.

[0012] Second, during the cyclic propagation process, reliability weights are generated based on the current frame features, propagation features, and motion information, so that reliable multi-frame complementary information is fully utilized, while unreliable propagation information is suppressed, thereby improving the temporal consistency and stability of video restoration.

[0013] Third, by introducing a frequency domain detail refinement module, the high-frequency response in the restored features is modeled and enhanced, which effectively improves the restoration effect of detailed areas such as edges and textures, and alleviates the problem of overly smooth restoration results.

[0014] Fourth, it can handle multiple degradation tasks such as video denoising, deblurring, and compression artifact removal within the same network framework. Compared with methods that design models separately for a single degradation type, it has better task adaptability and application flexibility.

[0015] Fifth, the reliability gating module and frequency domain detail refinement module have a simple structure and can be easily embedded into existing loop propagation video restoration frameworks. They can improve overall performance without significantly increasing the number of network parameters and computational complexity, making them easy to deploy in real-world scenarios. Attached Figure Description

[0016] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of an ALL-in-One video restoration method guided by a reliability gating mechanism according to an embodiment of the present invention; Figure 2 For the DeformAlignment module; Figure 3 For Motion Reliability Gated Module (MRG); Figure 4 For the Frequency Domain Detail Refinement (FDR) module; Figure 5 To recover mixed degraded video frames. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] Example 1 like Figure 1 As shown, this invention provides an ALL-in-One video restoration method guided by a reliability gating mechanism, comprising: Step S1: Acquire continuous low-quality video data; Step S2: Input continuous low-quality video data into the video restoration model, model the credibility of temporal propagation features through the motion reliability gating module, and enhance the edge and texture details of the restored video frames by combining the frequency domain / high-frequency detail refinement module, so as to achieve unified processing of multiple video restoration tasks such as video denoising, deblurring and compression artifact removal.

[0021] In one embodiment of the present invention, in step S2, the video restoration model is based on a recurrent propagation video restoration network, and introduces a motion reliability gating module (MRG) and a frequency domain detail refinement module (FDR) to improve the model's temporal alignment stability and detail recovery capability in various video degradation tasks. The video restoration model includes: an overall framework, a deformable alignment module, a motion reliability gating module (MRG), and a frequency domain detail refinement module (FDR).

[0022] The overall framework of this invention's video restoration model is used to achieve unified restoration for various video degradation tasks. The input is a continuous sequence of low-quality video frames, denoted as: Where t represents the number of input video frames, This represents the low-quality image of the i-th frame. The input sequence first passes through a shallow feature extraction module, mapping the original image to a high-dimensional feature space: Where E represents the shallow feature extraction function, and Fi represents the shallow feature corresponding to the i-th frame.

[0023] Subsequently, the network uses a motion estimation module to calculate the forward and backward optical flow between adjacent frames. The forward optical flow is used to propagate features from the previous frame to the next frame, and the backward optical flow is used to propagate features from the next frame to the previous frame. Where M represents the optical flow estimation network, and These represent the forward and backward motion information between adjacent frames, respectively.

[0024] After obtaining the optical flow, the network performs multiple rounds of cyclical propagation, for example, according to: The temporal features are propagated sequentially. In each round of propagation, the propagated features are first deformably aligned by the DeformAlignment module, and then combined with the reliability weights generated by the MRG module to suppress unreliable propagated information. After multiple rounds of propagation, the shallow features of the current frame are concatenated and fused with the features from each round of propagation, and then fed into the reconstruction module for image restoration. Finally, the FDR module enhances the high-frequency information in the reconstructed features and generates the final high-quality restored frame through the reconstruction output layer or the PixelShuffle upsampling layer, depending on the task type.

[0025] Furthermore, the DeformAlignment module spatially aligns features from adjacent frames during forward and backward propagation. Because video contains object motion, camera motion, occlusion, and local deformation, directly using features from adjacent frames can lead to spatial inconsistencies. Therefore, deformable alignment is needed to align the propagated features with the features of the current frame. Detailed module specifications are as follows... Figure 2 As shown.

[0026] Taking forward propagation as an example, let the current frame be the i-th frame, and the propagation feature of the previous frame be... Forward optical flow is First, optical flow is used to perform a preliminary spatial transformation on the propagation characteristics of the previous frame: Where w represents the spatial warping function based on optical flow. This indicates the propagation characteristics after coarse alignment.

[0027] Then, the current frame features Coarse alignment propagation characteristics The optical flow information is also stitched together to predict the offset and modulation mask required for deformable convolution. Here, concat represents the channel dimension concatenation operation.

[0028] The concatenated features are passed through an offset prediction network to obtain the offsets of deformable convolutions. and mask : Where D represents the prediction network consisting of multiple convolutional layers and activation functions, and Oi represents the predicted deformable convolutional offset. This represents the predicted modulation mask.

[0029] In this invention, the DeformAlignment module does not directly use the predicted offset and mask, but further introduces the Motion Reliability Gating (MRG) module to constrain them. The MRG generates a reliability weight map Ri based on the current frame features, the propagation features after coarse optical flow alignment, and the optical flow information, and applies this weight map to the offset and modulation mask of the deformable convolution to constrain the sampling offset and response intensity of unreliable regions before performing the deformable convolution.

[0030] in, This represents the offset after reliability gating constraints. This represents the modulation mask after reliability gating constraints. The Sigmoid is used to limit the mask value to the range [0,1].

[0031] Finally, the DeformAlignment module uses the constrained offset and modulation mask to perform deformable convolution on the initially aligned propagated features to obtain the final aligned features: Where DCN represents deformable convolution operation, This represents the propagation characteristics after alignment by the DeformAlignment module.

[0032] The backward propagation process is similar to the forward propagation process, except that its propagation direction is from the next frame to the previous frame.

[0033] By performing DeformAlignment in both forward and backward directions, the network can utilize information from both past and future frames, improving temporal consistency and detail enhancement during video restoration.

[0034] Furthermore, the MRG module is a lightweight gating structure used in this invention to estimate the reliability of propagation features. Its core function is to generate a spatially relevant reliability weight map based on the current frame features, propagation features, and motion information. This weight map is used to indicate whether the propagation features in different regions are reliable and to provide constraints for the offset and modulation mask in the DeformAlignment module. Detailed information about the module is as follows... Figure 3 As shown.

[0035] As shown in the figure, the MRG module receives three inputs: the shallow features Fi of the current frame, and the propagation features after preliminary optical flow transformation. And the corresponding optical flow information Vi. First, MRG calculates the absolute residual information between the current frame features and the propagation features: This residual information reflects the degree of consistency between the propagation features and the features of the current frame. When the difference between the two is large, it usually indicates that there may be motion estimation errors, occlusion, or degradation changes in the region.

[0036] Subsequently, the current frame features, propagation features, feature residual information, and motion amplitude information are concatenated along the channel dimension to form the gated input: in, This represents the intensity of optical flow in space. Regions with larger amplitude of motion usually correspond to rapid motion or complex deformation, and these regions are more prone to alignment errors.

[0037] Concatenated input features The data is fed into a lightweight convolutional gating network. This gating network first performs channel compression and information fusion through 1×1 convolutions, then extracts the local spatial context through 3×3 depthwise convolutions, and finally outputs a single-channel reliability response map through 1×1 convolutions. This process can be represented as: Here, G represents the lightweight convolution mapping function within the MRG.

[0038] Finally, the reliability response is normalized to the range [0,1] using the Sigmoid function to obtain the reliability weight graph: in, This represents the reliability weight map corresponding to the i-th frame. The closer the value is to 1, the more reliable the propagation characteristics of that spatial location are; the closer the value is to 0, the more likely there is occlusion, optical flow estimation error, feature misalignment, or erroneous propagation risk at that location.

[0039] In its implementation, the MRG module adopts a lightweight structure, where the number of hidden channels can be set to one-quarter of the number of input feature channels, with a minimum number of channels set to ensure expressive power. Through this design, MRG can effectively estimate the reliability of propagated features with relatively low parameter and computational requirements.

[0040] The reliability weight map output by the MRG module does not directly replace the current frame features or propagation features. Instead, it serves as a control signal for the DeformAlignment module, constraining the sampling offset and modulation mask in subsequent deformable convolutions. Thus, the network can enhance the alignment and utilization of propagation features in reliable regions and suppress missampling and erroneous responses in unreliable regions, thereby improving the stability of video restoration in complex motion scenes.

[0041] Furthermore, the FDR module is another core module of this invention, its function being to enhance high-frequency details such as edges and textures during the image reconstruction stage. In video restoration tasks, networks are prone to over-smoothing during denoising and deblurring, resulting in insufficient clarity of details and textures. Therefore, this invention introduces a frequency domain detail refinement module in the reconstruction stage to explicitly model the high-frequency responses in the reconstructed features. Detailed information about the module is as follows... Figure 4 As shown.

[0042] Let U be the intermediate feature output by the reconstruction module. i FDR first obtains the frequency domain detail response through a high-frequency extraction operator: Where Lap represents the Laplacian operator or other high-pass filtering function. This represents the high-frequency response extracted from the reconstructed features.

[0043] Subsequently, the high-frequency response is processed by a detail mapping network to obtain detail-enhanced features: in, This represents the detail mapping function, which consists of lightweight convolutional layers and activation functions.

[0044] To avoid over-enhancing noise or artifacts, the FDR module further generates detail adjustment weights: in, This represents the detailed weighted prediction function. This indicates that details are given increased weight.

[0045] Finally, residual fusion is performed between the detail enhancement features and the original reconstructed features: Here, β represents the detail enhancement intensity coefficient, which can be set as a fixed parameter or used as a learnable parameter in network training. After processing by the FDR module, the network can improve the sharpness of edges, textures, and local detail regions while maintaining the stability of the overall structure.

[0046] The specific process of the video restoration method of the present invention includes: 1. Input a batch of consecutive low-quality video frames, with the input shape being [batch={n}, number of frames={t}, channels={c}, resolution={h}×{w}].

[0047] 2. Perform shallow feature extraction on the input video frames to obtain high-dimensional shallow features, and change the shape to [batch={n}, number of frames={t}, channels={96}, resolution={h}×{w}] or set the corresponding resolution according to the specific task.

[0048] 3. Use an optical flow estimation network to calculate the forward and backward optical flow between adjacent frames. The forward optical flow is used for forward propagation, and the backward optical flow is used for backward propagation. The shape of the optical flow is [batch={n}, number of frames={t-1}, channels={2}, resolution={h}×{w}].

[0049] 4. The network enters the circular propagation phase, and multiple rounds of bidirectional feature propagation are carried out in the order of backward_1→forward_1→backward_2→forward_2.

[0050] 5. In each round of propagation, select the corresponding optical flow according to the propagation direction, and send the propagation characteristics of the previous or next moment into the DeformAlignment module for deformable alignment.

[0051] 6. The DeformAlignment module first uses optical flow to coarsely align the propagation features, and then uses a convolutional network with a reliability weight map generated by MRG to predict the offset and mask of the deformable convolution, finally obtaining the aligned propagation features.

[0052] 7. Input the current frame features, the propagation features after coarse optical flow alignment, the difference between the two, and the motion amplitude into the MRG module to generate a reliability weight map.

[0053] 8. Constrain the offset and modulation mask of deformable convolution based on the reliability weight map to ensure that reliable regions maintain effective sampling and response, while suppressing erroneous sampling and response in unreliable regions.

[0054] 9. The alignment propagation features are obtained by using the DeformAlignment module after reliability gating constraints. These features are then fused with the features of the current frame and refined using residual blocks to obtain the output features of the current propagation stage.

[0055] 10. After the image reconstruction module outputs intermediate reconstruction features, it sends them to the FDR module to extract high-frequency responses and perform detail enhancement.

[0056] 11. The features refined by FDR are fed into the final output layer to generate the restored video frames. The output shape is [batch={n}, number of frames={t}, channels={c}, resolution={h}×{w}].

[0057] The method of this invention has the following effect on the restoration of mixed degraded video frames: Figure 5 As shown, the left image is the input low-quality video frame, and the right image is the high-quality video frame restored by this method.

[0058] Example 2 This invention also provides an ALL-in-One video restoration system guided by a reliability gating mechanism, comprising: The first processing module is used to acquire continuous low-quality video data; The second processing module is used to input continuous low-quality video data into the video restoration model, model the credibility of temporal propagation features through the motion reliability gating module, and enhance the edge and texture details of the restored video frames by combining the frequency domain / high-frequency detail refinement module, so as to achieve unified processing of multiple video restoration tasks such as video denoising, deblurring and compression artifact removal.

[0059] As one embodiment of the present invention, the video restoration model is based on a cyclic propagation video restoration network, and introduces a motion reliability gating module (MRG) and a frequency domain detail refinement module (FDR) to achieve temporal alignment stability and detail recovery capability in various video degradation tasks.

[0060] As one embodiment of the present invention, the video restoration model includes: a DeformAlignment module, a Motion Reliability Gating Module (MRG), and a Frequency Domain Detail Refinement Module (FDR); wherein, the DeformAlignment module is used to spatially align the features of adjacent frames during forward and backward propagation; the MRG module is used to estimate the lightweight gating structure of propagation feature reliability and generate a spatially related reliability weight map based on the current frame features, propagation features, and motion information; and the FDR module is used to enhance edge and high-frequency texture details during the image reconstruction stage.

[0061] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. An ALL-in-One video restoration method guided by a reliability gating mechanism, characterized in that, include: Step S1: Acquire continuous low-quality video data; Step S2: Input continuous low-quality video data into the video restoration model, model the credibility of temporal propagation features through the motion reliability gating module, and enhance the edge and texture details of the restored video frames by combining the frequency domain / high-frequency detail refinement module, so as to achieve unified processing of multiple video restoration tasks such as video denoising, deblurring and compression artifact removal.

2. The ALL-in-One video restoration method based on a reliability gating mechanism as described in claim 1, characterized in that, The video restoration model is based on a cyclic propagation video restoration network, and introduces a motion reliability gating module (MRG) and a frequency domain detail refinement module (FDR) to achieve temporal alignment stability and detail recovery capability in various video degradation tasks.

3. The ALL-in-One video restoration method based on a reliability gating mechanism as described in claim 2, characterized in that, The video restoration model includes: a DeformAlignment module, a Motion Reliability Gating module (MRG), and a Frequency Domain Detail Refinement module (FDR). The DeformAlignment module is used to spatially align the features of adjacent frames during forward and backward propagation. The MRG module is used to estimate the reliability of propagation features using a lightweight gating structure, generating a spatially relevant reliability weight map based on the current frame features, propagation features, and motion information. The FDR module is used to enhance high-frequency details of edges and textures during the image reconstruction stage.

4. An ALL-in-One video restoration system guided by a reliability gating mechanism, characterized in that, include: The first processing module is used to acquire continuous low-quality video data; The second processing module is used to input continuous low-quality video data into the video restoration model, model the credibility of temporal propagation features through the motion reliability gating module, and enhance the edge and texture details of the restored video frames by combining the frequency domain / high-frequency detail refinement module, so as to achieve unified processing of multiple video restoration tasks such as video denoising, deblurring and compression artifact removal.

5. The ALL-in-One video restoration system based on a reliability gating mechanism as described in claim 4, characterized in that, The video restoration model is based on a cyclic propagation video restoration network, and introduces a motion reliability gating module (MRG) and a frequency domain detail refinement module (FDR) to achieve temporal alignment stability and detail recovery capability in various video degradation tasks.

6. The ALL-in-One video restoration system based on a reliability gating mechanism as described in claim 5, characterized in that, The video restoration model includes: a DeformAlignment module, a Motion Reliability Gating module (MRG), and a Frequency Domain Detail Refinement module (FDR). The DeformAlignment module is used to spatially align the features of adjacent frames during forward and backward propagation. The MRG module is used to estimate the reliability of propagation features using a lightweight gating structure, generating a spatially relevant reliability weight map based on the current frame features, propagation features, and motion information. The FDR module is used to enhance high-frequency details of edges and textures during the image reconstruction stage.

Images