Video restoration method and system

Abstract

The application relates to a video restoration method, comprising the following steps: pre-training a generative video restoration network based on large-scale high-quality video data to obtain a basic model with strong detail generation capability; constructing a reference image set and extracting an attribute set through an attribute extractor; preparing a training target high-definition video corresponding to the reference image, synthesizing a low-definition video and calculating a supervision loss; freezing the basic model weight obtained through pre-training to perform restoration under attribute condition constraints; inputting a real low-definition video and a target object attribute set to generate a high-resolution video and realize consistent industrial-level restoration output of the target object; and the application also relates to a video restoration system. The application can keep the consistency of target object attributes in cross-lens and long-time sequence segments while keeping high perceptual quality.

Description

Technical Field

[0001] This invention relates to a video restoration method and system. Background Technology

[0002] Low-definition videos suffer from inherent information loss due to degradation caused by compression, downsampling, noise, blurring, and bitrate jitter. This results in inherent ill-posedness in modeling low-definition videos for video restoration (including video super-resolution, denoising, deblurring, and decompression artifact removal).

[0003] Video restoration aims to recover high-quality videos from low-quality ones. Common tasks include video super-resolution, denoising, deblurring, and decompression artifact removal. In recent years, generative methods (especially diffusion models) have been introduced into video restoration, significantly improving perceptual quality and detail richness by filling in missing details in a high-dimensional generative space.

[0004] Current generative video restoration (such as diffusion model-driven restoration) has achieved excellent performance in terms of perceptual quality. However, in industrial-grade film and television restoration scenarios, maintaining the consistency of classic "target objects" such as characters, costumes, props or buildings under shot transitions, long time sequences and complex movements remains a major challenge: textures / identifiers that do not match the original film may be generated, causing identity, clothing patterns and architectural details to drift or flicker in the time dimension.

[0005] Specifically, existing generative video restoration methods tend to generate object details that do not semantically match the original video, and it is difficult to ensure the consistency of identity / appearance of the same object (person, clothing logo, architectural texture, etc.) within shot changes or long segments. Common problems include detail drift, flickering, and erroneous texture "illusions." Currently, there is a lack of video restoration methods that can maintain target consistency. To maintain target consistency, a common approach is to introduce reference images as supplements for lost information. However, reference-based restoration mainly focuses on the image domain; and it usually requires that the reference image be in the same scene / viewpoint as the input and have a high degree of overlap, which is difficult to meet in scenarios such as old movies / historical figures and buildings; moreover, the granularity of consistency control is usually at the whole image level, making it difficult to achieve fine-grained consistency and controllable restoration for specific objects. Summary of the Invention

[0006] In view of this, it is necessary to provide a video restoration method and system that can maintain the consistency of target object attributes across shots and long time sequences, while maintaining high perceptual quality.

[0007] This invention provides a video restoration method, which includes: Step S1, pre-training a generative video restoration network based on large-scale high-quality video data to obtain a basic model with strong detail generation capabilities; Step S2, constructing a reference image set and extracting an attribute set through an attribute extractor; preparing a high-definition video of the training target corresponding to the reference image, synthesizing a low-definition video and calculating the supervised loss; freezing the weights of the pre-trained basic model and adding a low-rank adapter for fine-tuning to perform restoration under attribute constraints; Step S3, inputting a real low-definition video and the attribute set of the target object to generate a high-resolution video, achieving an industrial-grade restoration output consistent with the target object.

[0008] Step S1 includes: Constructing a generative video restoration network Its input is low-definition video. The output is a restored video. : .

[0009] Step S1 includes: A pre-trained text-based video pedigree model is used, which includes a three-dimensional variational autoencoder encoder E, a decoder D, and a DiT diffusion denoising network.

[0010] Step S1 includes: Construct generative modeling, where the diffusion process is defined as the data distribution Compared with standard normal noise distribution A linear path between them; given a clean sample of latent variables. and its time step Noise correspondence on The diffusion input obtained from a single calculation in forward diffusion for:

[0011] The denoising process is defined by solving an ordinary differential equation (ODE) from the noise distribution. To data distribution Inverse mapping:

[0012] in For the weights of the diffusion network Parameterized velocity field, The text prompt conditions are represented; the final training objective uses conditional flow matching to regress velocity under given conditions. Its definition is as follows: .

[0013] Step S2 includes: Step S21: Construct a reference image set and prepare high-definition videos of the training target; Step S22, introduce low-rank adapter parameters , attribute conditions With low-definition video Using splicing as a conditional input, a video restoration mapping with reference conditions is obtained; Step S23: Freeze the pre-trained weights, add a low-rank adapter for fine-tuning, and achieve video restoration guided by reference attribute conditions.

[0014] Step S21 includes: Step S211: Construct a reference image set R and obtain an attribute set using an attribute extractor. in For single-modal or single-granularity conditions, the attribute extractor E(·) is given based on the attribute target to be restored; Step S212: Prepare the high-definition video X corresponding to the reference image as the training target, which will be used to synthesize the low-definition video as training input. And it is used to monitor loss calculation.

[0015] Step S22 includes: In the basic model Several attention / linear layers introduce low-rank adapter LoRA parameters. :

[0016] Where W represents the frozen pre-trained base weights, and only low-rank matrices A and B are trained during the training phase; the attribute conditions are... With low-definition video Using concatenation as a conditional input, we obtain a video restoration map with reference conditions:

[0017] For high-definition video that matches the target predicted by the model, target consistency is determined by the attribute set. This is achieved by guiding the generative restoration process.

[0018] Step S23 includes: Freeze pre-trained weights and utilize a small set of attributes To optimize the LoRA parameter Δ during the adaptation phase, the objective function is:

[0019] The final optimized adapter weight Δ is the result of adapting to the attribute set. Specific weights are used to adapt the desired fit to It can restore any video clip.

[0020] Step S3 includes: Given a real low-resolution video Restoring the execution conditions with respect to the target attribute set A:

[0021] Output To achieve a high-definition restored video that is consistent with the final target object.

[0022] This invention provides a video restoration system, which includes a pre-training module, an adaptation module, and a restoration output module, wherein: The pre-training module is used to pre-train a generative video restoration network based on large-scale high-quality video data to obtain a basic model with strong detail generation capabilities. The adaptation module is used to construct a reference image set and extract an attribute set through an attribute extractor; prepare high-definition videos of training targets corresponding to the reference images, synthesize low-definition videos and calculate supervised loss; freeze the weights of the pre-trained basic model, add a low-rank adapter for fine-tuning, so as to restore the model under attribute constraints. The restoration output module is used to input a real low-definition video and a set of target object attributes to generate a high-resolution video, achieving industrial-grade restoration output consistent with the target object.

[0023] The beneficial effects of this application include: (1) Stronger consistency: Compared with generative restoration that relies solely on the input video, this invention introduces “target object attribute conditions” and generates the process by using a few samples to adapt explicit constraints, thereby significantly reducing object detail drift and flickering and improving cross-shot consistency.

[0024] (2) More relaxed reference requirements: Compared with the traditional reference restoration, which requires the reference and input to be in the same scene / view, the reference image of this invention is used to extract attributes rather than migrate pixel by pixel, which is suitable for old movies, historical figures / buildings, etc. where it is difficult to obtain high-definition references in the same scene.

[0025] (3) Low training cost and deployment friendly: Only a small number of parameters are updated through the low-rank adapter, which can quickly complete object adaptation under the condition of few samples; during deployment, only the basic model and adapter weights are loaded to complete the restoration.

[0026] (4) High scalability: The attribute extractor and attribute consistency loss can be extended to different object types and different attribute targets (texture, style, semantic tags, etc.) according to industrial needs, improving the versatility of the solution.

[0027] (5) Finer granularity and stronger controllability of object consistency: Compared with the existing reference restoration technology, which mostly uses "whole image / global texture migration" as consistency constraint, the present invention uses the target object attribute set as a condition for constraint, which can achieve fine-grained consistency maintenance at the object level or even at the object local level (such as facial features, clothing patterns / patterns, building facade components, etc.); and by selecting different attribute extractors and object area constraint methods, controllable restoration of consistency dimensions such as "identity / appearance / texture / style" can be achieved, reducing the risk of non-target areas being mistakenly migrated or contaminated by generative "illusion". Attached Figure Description

[0028] Figure 1 This is a flowchart of the video restoration method of the present invention; Figure 2 This is a schematic diagram of the reference video restoration model architecture in an embodiment of the present invention; Figure 3 This is a hardware architecture diagram of the video restoration system of the present invention. Detailed Implementation

[0029] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0030] Symbols and definitions: High-definition input video containing N frames ; The degradation operator D(·) is used to synthesize low-resolution degradation. ; Target output high-definition video ; Reference Image Set ; Attribute extractor E(·); Output the set of attributes of the target object A: A = E(R).

[0031] See Figure 1 The diagram shown is a flowchart of a preferred embodiment of the video restoration method of the present invention.

[0032] Step S1: Pre-training the generative video restoration network. That is, pre-training the generative video restoration network based on large-scale, high-quality video data to obtain a foundational model with strong detail generation capabilities. Specifically: Constructing a generative video restoration network Its input is low-definition video. The output is a restored video. : In this embodiment, A pre-trained text-based video pedestal model is employed, whose architecture includes a 3D variational autoencoder (3D VAE) encoder E, a decoder D, and a DiT diffusion denoising network. Low-resolution video input is used as a condition to model generative video restoration tasks. The training objective utilizes a text-based video pedestal model based on the Rectified-Flow framework to construct generative modeling, where the diffusion process is defined as the data distribution... Compared with standard normal noise distribution A linear path between them. Given a clean sample of latent variables. and its time step Noise correspondence on The diffusion input obtained from a single calculation in forward diffusion for:

[0033] The denoising process is defined by solving an ordinary differential equation (ODE) from the noise distribution. To data distribution Inverse mapping:

[0034] in For the weights of the diffusion network Parameterized velocity field, This indicates the text prompt conditions. The final training objective employs Conditional Flow Matching (CFM) to regress speed under given conditions. Its definition is as follows:

[0035] Step S2, Few-shot Object Adaptation (Low-Rank Adapter Training). This involves constructing a reference image set and extracting attribute sets using an attribute extractor; preparing high-resolution videos of the training targets corresponding to the reference images; synthesizing low-resolution videos and calculating supervised loss; freezing the pre-trained base model weights; and adding a low-rank adapter for fine-tuning to achieve restoration under attribute constraints. Please refer to [link to relevant documentation]. Figure 2 Specifically, it includes: Step S21, constructing a reference image set and preparing high-definition videos for training targets, including: Step S211: Construct a reference image set R and obtain an attribute set using an attribute extractor. .in For single-modal or single-granularity conditions, the attribute extractor E(·) is given based on the target attribute to be restored. Specifically, it includes: Transferring object-level attributes (instance mask, key points, local texture regions) from the reference image: E(·) is the segmentation model or key point detector, extracting attributes from the reference image as the corresponding granular objects to be segmented; The style attributes (architectural style, color, material, brushstroke) of the reference image set are transferred. E(·) is an abstract style extractor that extracts the unified style attributes of the reference image set.

[0036] Step S212: Prepare the high-definition video X corresponding to the reference image as the training target, which will be used to synthesize the low-definition video as training input. And it is used to monitor loss calculation.

[0037] Step S22, Conditional Restoration and Low-Rank Adapter In the basic model Several attention / linear layers introduce low-rank adapter (LoRA) parameters. :

[0038] Where W represents the frozen pre-trained base weights, and only low-rank matrices A and B are trained during the training phase. The attribute conditions are... With low-definition video Using concatenation as a conditional input, we obtain a video restoration map with reference conditions:

[0039] For high-definition video that matches the target predicted by the model, target consistency is determined by the attribute set. This is achieved by guiding the generative restoration process.

[0040] Step S23, Training Objective and Loss Design Freeze the pre-trained weights, add a low-rank adapter parameter Δ for fine-tuning, and utilize a few-sample attribute set. To adapt and optimize the parameter Δ, the objective function is:

[0041] The final optimized adapter weight Δ is the result of adapting to the attribute set. Specific weights are used to adapt the desired fit to It can restore any video clip.

[0042] Step S3 involves inputting a real low-resolution video and a set of target object attributes. High-resolution video is generated through conditional restoration using a "base model + LoRA" approach, achieving industrial-grade restoration output consistent with the target object. Specifically: Given a real low-resolution video With the target attribute set A (extracted from the reference image set R), the execution conditions are restored:

[0043] Output To achieve a high-definition restored video that is consistent with the final target object.

[0044] See Figure 3 The diagram shown is a hardware architecture diagram of the video restoration system 10 of the present invention. The system includes: a pre-training module 101, an adaptation module 102, and a restoration output module 103.

[0045] The pre-training module 101 is used to pre-train the generative video restoration network. That is, it pre-trains the generative video restoration network based on large-scale, high-quality video data to obtain a base model with strong detail generation capabilities. Specifically: Constructing a generative video restoration network Its input is low-definition video. The output is a restored video. : In this embodiment, A pre-trained text-based video pedestal model is employed, whose architecture includes a 3D variational autoencoder (3D VAE) encoder E, a decoder D, and a DiT diffusion denoising network. Low-resolution video input is used as a condition to model generative video restoration tasks. The training objective utilizes a text-based video pedestal model based on the Rectified-Flow framework to construct generative modeling, where the diffusion process is defined as the data distribution... Compared with standard normal noise distribution A linear path between them. Given a clean sample of latent variables. and its time step Noise correspondence on The diffusion input obtained from a single calculation in forward diffusion for:

[0046] The denoising process is defined by solving an ordinary differential equation (ODE) from the noise distribution. To data distribution Inverse mapping:

[0047] in For the weights of the diffusion network Parameterized velocity field, This indicates the text prompt conditions. The final training objective employs Conditional Flow Matching (CFM) to regress speed under given conditions. Its definition is as follows:

[0048] The adaptation module 102 is used for few-shot object adaptation (low-rank adapter training). That is, it constructs a reference image set and extracts attribute sets using an attribute extractor; prepares high-definition videos of the training targets corresponding to the reference images, synthesizes low-definition videos and calculates supervised loss; freezes the pre-trained base model weights, and adds a low-rank adapter for fine-tuning to restore the model under attribute constraints. Please refer to [link to relevant documentation]. Figure 3 Specifically, it includes: First, construct a reference image set and prepare high-definition videos for the training targets, including: Construct a reference image set R, and obtain the attribute set using an attribute extractor. .in For single-modal or single-granularity conditions, the attribute extractor E(·) is given based on the target attribute to be restored. Specifically, it includes: Transferring object-level attributes (instance masks, keypoints, local texture regions) from a reference image: E(·) represents the segmentation model or keypoint detector, extracting attributes from the reference image to represent the corresponding granular objects in the segmentation. See [link to relevant documentation]. Figure 2 ; The style attributes (architectural style, color, material, brushstroke) of the reference image set are transferred. E(·) is an abstract style extractor that extracts the unified style attributes of the reference image set.

[0049] Prepare a high-resolution video X corresponding to the reference image as the training target, which will be used to synthesize a low-resolution video as training input. And it is used to monitor loss calculation.

[0050] Next, conditional restoration and low-rank adapter: In the basic model Several attention / linear layers introduce low-rank adapter (LoRA) parameters. :

[0051] Where W represents the frozen pre-trained base weights, and only low-rank matrices A and B are trained during the training phase. The attribute conditions are... With low-definition video Using concatenation as a conditional input, we obtain a video restoration map with reference conditions:

[0052] For high-definition video that matches the target predicted by the model, target consistency is determined by the attribute set. This is achieved by guiding the generative restoration process.

[0053] Finally, the training objective and loss design: Freeze the pre-trained weights, add a low-rank adapter parameter Δ for fine-tuning, and utilize a few-sample attribute set. To adapt and optimize the parameter Δ, the objective function is:

[0054] The final optimized adapter weight Δ is the result of adapting to the attribute set. Specific weights are used to adapt the desired fit to It can restore any video clip.

[0055] The restoration output module 103 is used to input a real low-resolution video and a set of target object attributes, and generates a high-resolution video through conditional restoration using a "base model + LoRA" approach, achieving industrial-grade restoration output consistent with the target object. Specifically: Given a real low-resolution video With the target attribute set A (extracted from the reference image set R), the execution conditions are restored:

[0056] Output The final high-definition restored video is consistent with the target object.

[0057] In other embodiments of the present invention: In addition to low-rank adapters, prefix tuning, adapter layers, and partial layer unfreezing fine-tuning can also be used. Object-level attributes can be replaced by keypoint / pose estimation instead of segmentation; style attributes can be replaced by color histograms / material encoding instead of deep style extraction; semantic attributes can be replaced by text prompts and visual-language model encoding. Build attribute sets for multiple objects and apply multi-condition joint constraints; or simultaneously constrain the same object with "identity + clothing texture + style"; Video representation can be video in pixel space, video latent variables encoded into latent space by an encoder, or video tokens that have undergone transformation.

[0058] In addition to video restoration, this invention can be extended to scenarios that require object consistency, such as animation remake, old documentary restoration, low bitrate live broadcast playback enhancement, and cultural relic image restoration.

[0059] This invention provides generative video restoration with "object-level consistency": it constructs a set of few-sample reference images and extracts the target object attributes as conditions, and trains a pre-trained generative video restoration network with a low-rank adapter (LoRA) to guide the restoration process by attribute conditions, thereby maintaining the consistency of target object attributes across shots and long time sequences, while maintaining the high perceptual quality of generative methods.

[0060] Although the present invention has been described with reference to the present preferred embodiments, those skilled in the art should understand that the above preferred embodiments are only used to illustrate the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A video restoration method, characterized in that, The method includes: Step S1: Pre-train a generative video restoration network based on large-scale, high-quality video data to obtain a basic model with strong detail generation capabilities. Step S2: Construct a reference image set and extract the attribute set through an attribute extractor; prepare high-definition videos of the training targets corresponding to the reference images, synthesize low-definition videos and calculate supervised loss; freeze the weights of the pre-trained basic model, add a low-rank adapter for fine-tuning, so as to restore the model under attribute constraints. Step S3: Input the real low-definition video and the target object attribute set, generate a high-resolution video, and achieve industrial-grade restoration output consistent with the target object.

2. The method as described in claim 1, characterized in that, Step S1 includes: Constructing a generative video restoration network Its input is low-definition video. The output is a restored video. : .

3. The method as described in claim 2, characterized in that, Step S1 includes: A pre-trained text-based video pedigree model is used, which includes a three-dimensional variational autoencoder encoder E, a decoder D, and a DiT diffusion denoising network.

4. The method as described in claim 3, characterized in that, Step S1 includes: Construct generative modeling, where the diffusion process is defined as the data distribution Compared with standard normal noise distribution A linear path between them; given a clean sample of latent variables. and its time step Noise correspondence on The diffusion input obtained from a single calculation in forward diffusion for: The denoising process is defined by solving an ordinary differential equation (ODE) from the noise distribution. To data distribution Inverse mapping: in For the weights of the diffusion network Parameterized velocity field, The text prompt conditions are represented; the final training objective uses conditional flow matching to regress velocity under given conditions. Its definition is as follows: 。 5. The method as described in claim 4, characterized in that, Step S2 includes: Step S21: Construct a reference image set and prepare high-definition videos of the training target; Step S22, introduce low-rank adapter parameters , attribute conditions With low-definition video Using splicing as a conditional input, a video restoration mapping with reference conditions is obtained; Step S23: Freeze the pre-trained weights, add a low-rank adapter for fine-tuning, and achieve video restoration guided by reference attribute conditions.

6. The method as described in claim 5, characterized in that, Step S21 includes: Step S211: Construct a reference image set R and obtain an attribute set using an attribute extractor. in For single-modal or single-granularity conditions, the attribute extractor E(·) is given based on the attribute target to be restored; Step S212: Prepare the high-definition video X corresponding to the reference image as the training target, which will be used to synthesize the low-definition video as training input. And it is used to monitor loss calculation.

7. The method as described in claim 6, characterized in that, Step S22 includes: In the basic model Several attention / linear layers introduce low-rank adapter LoRA parameters. : Where W represents the frozen pre-trained base weights, and only low-rank matrices A and B are trained during the training phase; the attribute conditions are... With low-definition video Using concatenation as a conditional input, we obtain a video restoration map with reference conditions: For high-definition video that matches the target predicted by the model, target consistency is determined by the attribute set. This is achieved by guiding the generative restoration process.

8. The method as described in claim 7, characterized in that, Step S23 includes: Freeze pre-trained weights and utilize a small set of attributes To optimize the LoRA parameter Δ during the adaptation phase, the objective function is: The final optimized adapter weight Δ is the result of adapting to the attribute set. Specific weights are used to adapt the desired fit to It can restore any video clip.

9. The method as described in claim 8, characterized in that, Step S3 includes: Given a real low-resolution video Restoring the execution conditions with respect to the target attribute set A: Output To achieve a high-definition restored video that is consistent with the final target object.

10. A video restoration system, characterized in that, The system includes a pre-training module, an adaptation module, and a recovery output module, wherein: The pre-training module is used to pre-train a generative video restoration network based on large-scale high-quality video data to obtain a basic model with strong detail generation capabilities. The adaptation module is used to construct a reference image set and extract an attribute set through an attribute extractor; prepare high-definition videos of training targets corresponding to the reference images, synthesize low-definition videos and calculate supervised loss; freeze the weights of the pre-trained basic model, add a low-rank adapter for fine-tuning, so as to restore the model under attribute constraints. The restoration output module is used to input a real low-definition video and a set of target object attributes to generate a high-resolution video, achieving industrial-grade restoration output consistent with the target object.

Images