Face video recovery method for feedback learning guidance in end-cloud collaborative environment

Abstract

The invention discloses a face video recovery method based on feedback learning guidance in an end-cloud collaborative environment. The method comprises two training stages: 1, in a pre-training stage of a cloud side face video recovery model, analyzing a high-definition identity reference image into a prompt of a text and a visual mode, guiding a de-noising recovery process of the model, and solving a problem of result identity averaging caused by lack of identity context information; 2, in an end-cloud collaborative learning stage, performing feedback learning on each end side in combination with a face similarity score and a face natural score, so that a recovery result gives consideration to both identity fidelity and human preference; in order to generate a face natural score reflecting the preference of human to the natural degree of a recovery result, a corresponding reward model is trained in a human annotation preference data set; according to the method, an index mixed sampling strategy is provided in the reasoning stage to solve the problem of inter-frame identity drift.

Description

Technical Field

[0001] The present invention belongs to the fields of computer vision and machine learning, and specifically relates to a face video restoration method guided by feedback learning in an end-cloud collaborative environment. Background Art

[0002] With the widespread use of video content, face video restoration technology has become an important research direction in the field of computer vision, especially in the areas of video super-resolution and denoising. Traditional restoration methods mostly rely on single image restoration techniques, such as image denoising and super-resolution restoration. However, these methods often struggle to cope with complex face video restoration tasks, especially in low-resolution and high-noise environments, and struggle to preserve high-quality video details and authentic identity features.

[0003] A key challenge in face restoration from video is maintaining the stability and consistency of the person's identity in the video. In low-resolution videos, due to the lack of identity context, the restoration results often suffer from identity distortion or fluctuation, failing to accurately recover the identity features from the original video. To address this challenge, some face restoration methods have introduced reference priors, leveraging high-quality reference images of the same identity, which theoretically can achieve high fidelity. However, in practice, accurately transferring identity information from low-quality images to reference images remains challenging due to differences in pose, lighting, and semantics between the low-quality images and the reference images.

[0004] In addition, another important challenge facing face video restoration is the distribution discrepancy between training data and actual scene data. This distribution inconsistency may lead to suboptimal restoration results when the model is used in real-world application environments. To address this issue, end-cloud collaborative computing has gradually become an effective strategy for improving face video restoration performance in recent years. In the end-cloud collaborative computing framework, the cloud leverages its powerful computing resources to train and globally optimize large-scale models, while end-side devices use local computing power for real-time inference, dynamic adjustment, and online learning. Through this collaborative model, the cloud can perform global optimization based on a wide range of training data, while end-side devices can quickly adapt based on real-time collected data, effectively bridging the distribution discrepancy between training data and actual application scenarios, and improving the accuracy and robustness of the restoration effect.

[0005] Specifically, the shortcomings of the existing technology are:

[0006] (1) Identity consistency problem: Traditional face video restoration methods are difficult to effectively maintain the identity consistency of people in low-resolution videos, resulting in identity distortion or fluctuation in the restoration results, and are unable to accurately restore the identity features in the original video.

[0007] (2) Distribution differences between training data and application scenarios: There are distribution differences between the image datasets used in training existing models and real application scenarios, which leads to a significant decrease in the recovery effect of the model in actual applications and cannot meet the needs of high-quality recovery. Summary of the Invention

[0008] In order to solve the problems in the background technology, the present invention provides a face video restoration method guided by feedback learning in an end-cloud collaborative environment.

[0009] In order to achieve the above-mentioned object of the invention, the present invention specifically adopts the following technical solutions:

[0010] 1. Feedback-guided face video restoration method in a device-cloud collaborative environment

[0011] The specific steps include:

[0012] 1) Build a face video restoration model based on the Face2Text encoder, face vision encoder, and video restoration base model;

[0013] 2) During the pre-training phase of the cloud-side face video restoration model, the high-definition identity reference image is parsed into textual and visual modality cues, and a decoupled cross-attention mechanism is used to guide the model's denoising and restoration process. After pre-training is complete, the resulting model weights are distributed to each client.

[0014] 3) During the device-cloud collaborative learning phase, each device-side model performs feedback learning based on identity preservation loss and human preference loss. After each device-side model completes parameter optimization, the optimized model parameters are uploaded to the cloud for aggregation and initialization of parameters required for the next round of communication.

[0015] 4) In the inference stage of the face video restoration model, multiple inferences are performed on the low-quality video to be restored, and the restored video with the highest reward score is output as the output result.

[0016] The terminal side device includes a computer, and the cloud side device includes a server.

[0017] In step 2), the data for the supervised training process includes high-quality videos, corresponding low-quality videos, and identity reference images.

[0018] The step 2) specifically includes the following steps:

[0019] 2.1) Generate multimodal cues using high-definition identity reference images, specifically textual and visual cues;

[0020] in:

[0021] The text modality hint is obtained by passing the pre-trained Face2Text encoder through the reference image, denoted as c text The Face2Text encoder consists of a facial attribute detector, a large language model, and a CLIP text encoder connected in sequence;

[0022] The visual modality hint is generated by the pre-trained face visual encoder, denoted as c img ;

[0023] 2.2) Using a decoupled cross-attention mechanism to perform multimodal cue control on the denoising process of the face video restoration model M;

[0024] The face video restoration model M is composed of multiple U-Net blocks. Let Z be the input of the U-Net block. Cross-attention is performed on the text and visual modalities respectively and summed up to obtain the decoupled cross-attention output Z′. The process is expressed as:

[0025] Z′=Attention(Q,K t , V t )+λ.Attention(Q,K i , V i )

[0026] Among them, the matrix Q = Z..W q ,

[0027] Among them, (Q, K t , V t ) are the query matrix, key matrix and value matrix of the text cross attention operation; K i ,V i is the key matrix and value matrix of the visual cross-attention operation; W q , is the learnable weight;

[0028] 2.3) Joint High-Resolution Video Reconstruction Loss and identity retention loss Pre-train the cloud-side face video restoration model, and the overall loss is recorded as:

[0029]

[0030] In the step 2.3):

[0031] High-resolution video reconstruction loss The prediction loss for the video diffusion model is expressed as:

[0032]

[0033] Among them, c lqRepresents a low-resolution video, and controls the denoising process through ControlNet; c text c img The denoising process is controlled by the decoupled cross-attention mechanism in step 2.2); z is the encoding obtained by inputting the high-resolution video into the Encoder of the variational autoencoder, z t is the result of the forward process of the diffusion model gradually adding noise to the t step, ∈ is the added noise, ∈ θ (.) is the noise predictor of the face video restoration model M, and θ is the corresponding learnable parameter;

[0034] Loss of identity preservation Expressed as:

[0035]

[0036] Among them, cos_sim(.) is used to calculate f ref and The cosine similarity of f ref is the face visual encoding of the reference identity image, denoted as f ref =FaceEnc(ref); is the face visual encoding of the i-th key frame of the generated video, expressed as FaceEnc(.) is a face visual encoder.

[0037] The step 3) specifically comprises the following steps:

[0038] 3.1) Identity retention loss in each end-side model and human preference loss For feedback learning, the training loss is as follows:

[0039]

[0040] Among them, ζ is a hyperparameter used to adjust the ratio of the two losses;

[0041] 3.2) Each end uploads the updated face video recovery model M i The parameter θ i , the cloud-side server aggregates these updated parameters as the initial model parameters for the next round of communication. The specific formula is as follows:

[0042]

[0043] Where N is the number of devices on the end side.

[0044] The human preference loss By the pre-trained reward model g ψ Get, specifically expressed as:

[0045]

[0046] Among them, x gen Videos generated by the face video restoration model;

[0047] Reward Model g ψ Videos can be scored and reflect human preferences for the naturalness of the restored results. Reward model g ψ It is trained on a self-collected human-annotated preference dataset, and its training loss is expressed as:

[0048]

[0049] Among them, g ψ (x i ,c text , c img ) indicates that the video x i , text modal prompt of reference identity c text , visual modal prompt c img When it is input, the reward model g ψ Output reward score.

[0050] Among them, x i and x j is the data used to train the reward model, x i It is x j Videos that are more in line with human aesthetic preferences.

[0051] In step 3): in the unsupervised training process of end-cloud collaborative learning, the data used includes low-quality videos and identity reference images.

[0052] In step 4), an exponential mixing sampling strategy is proposed in the inference stage to deal with the inter-frame identity drift problem that occurs during long video restoration;

[0053] The sampling strategy of the exponential mixing is specifically as follows:

[0054] In order to achieve inference acceleration, the input low-quality video is segmented into multiple fixed-length video segments. The hidden layer encoding of the segmented video segments is mixed to enhance the consistency of identity between frames, which is expressed as:

[0055]

[0056] Among them, z t(i) represents the hidden layer code of the i-th video clip at time step t. The superscript in the formula indicates the range of frame numbers, stack(.) represents the vector stacking operation, O is the shared frame length of adjacent video clips, and F is the frame length of each video clip. j is the j-th video clip.

[0057] 2. A terminal

[0058] including memory and processor;

[0059] The memory is used to store computer programs;

[0060] The processor is used to implement the above-mentioned feedback learning guided face video restoration method in the end-cloud collaborative environment when executing the computer program.

[0061] 3. A computer-readable storage medium

[0062] The storage medium stores a computer program, and when the computer program is executed by the processor, the face video restoration method guided by feedback learning in the above-mentioned end-cloud collaborative environment is implemented.

[0063] Beneficial effects of the present invention:

[0064] This paper addresses the issue of missing identity context in face video restoration tasks by using high-definition identity reference images to generate multimodal cues and controlling the denoising process of the face video restoration model through a decoupled cross-attention mechanism. Furthermore, in a collaborative end-to-end cloud environment, this paper employs identity-preserving loss and human preference loss for feedback learning, effectively bridging the distribution gap between training data and real-world scene data, resulting in more accurate and robust restoration results in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 This is a flowchart of the pre-training of the face video restoration model of the present invention;

[0066] Figure 2 A schematic diagram of feedback learning in a device-cloud collaborative environment according to the present invention;

[0067] Figure 3 A comparison of the effects of the model trained by the present invention and the most advanced face restoration model at different levels of degradation; DETAILED DESCRIPTION

[0068] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0069] like Figure 1 and Figure 2 As shown, the face video restoration method guided by feedback learning in the end-cloud collaborative environment of the present invention includes the following steps:

[0070] 1) During the pre-training phase of the cloud-side face video restoration model, steps 1.1) to 1.3) are performed to parse the high-definition identity reference image into textual and visual modality cues. A decoupled cross-attention mechanism is then used to guide the model's denoising and restoration process. The supervised training data includes the high-resolution video, the corresponding low-resolution video, and the identity reference image.

[0071] 1.1) Generate multimodal prompts using high-definition identity reference images. The text modality prompt is obtained by passing the reference image through the pre-trained face attribute detector, large language model, and CLIP text encoder in sequence, denoted as c text ; The visual modality hint is generated by the pre-trained face encoder, denoted as c img .

[0072] 1.2) Use the decoupled cross-attention mechanism to perform multimodal prompt control on the denoising process of the face video restoration model M. The face video restoration model M is composed of multiple U-Net blocks, with Z as the input of the U-Net block. The decoupled cross-attention output Z′ is obtained by performing cross-attention on the text and visual modalities and summing them. The process is expressed as

[0073] Z′=Attention(Q,K t , V t )+λ.Attention(Q,K i ,V i ),

[0074] Among them, the matrix Q = ZW q ,

[0075] (Q,K t ,V t ) are the query matrix, key matrix and value matrix of the text cross attention operation. i , V i Used for visual cross-attention.

[0076] 1.3) Joint high-resolution video reconstruction loss and identity retention loss Pre-train the cloud-side face video restoration model, and the overall loss is recorded as

[0077]

[0078] The high-resolution video reconstruction loss The prediction loss for the video diffusion model is expressed as:

[0079]

[0080] c lq Represents a low-resolution video, and controls the denoising process through ControlNet. text , c img The denoising process is controlled by the above-mentioned decoupled cross-attention mechanism. z is the encoding obtained by inputting the high-resolution video into the Encoder of the variational autoencoder, z t is the result of the forward process of the diffusion model gradually adding noise to the t step, ∈ is the added noise, where ∈ θ (.) is the noise predictor of the face video restoration model M, and θ is the corresponding learnable parameter.

[0081] in, is the identity preservation loss, denoted as cos_sim(.) is used to calculate the cosine similarity of two face visual codes. ref is the face visual encoding of the reference identity image, denoted as f ref =FaceEnc(ref); is the face visual encoding of the i-th key frame of the generated video, expressed as FaceEnc(.) is a face visual encoder.

[0082] 2) During the device-cloud collaborative learning phase, steps 2.1) to 2.2) are executed. Each device-side model performs feedback learning based on the identity preservation loss and human preference loss. The unsupervised training process uses data including low-resolution video and identity reference images. After each device-side model completes parameter optimization, the optimized model parameters are uploaded to the cloud for aggregation and initialization of parameters required for the next round of communication.

[0083] 2.1) If Figure 2 As shown, the model on each side retains the identity loss and human preference loss For feedback learning, the training loss is as follows:

[0084]

[0085] Human preference loss By the reward model g ψ The model can score videos and reflect human preferences for the naturalness of the restored results. The model is trained on a self-collected human annotation preference dataset, and its training loss is expressed as:

[0086]

[0087] g ψ (x i , ctext , c img ) represents a given video x i and textual and visual modal cues of reference identity c text and c img When the reward model g ψ The reward score, while human preference loss

[0088] 2.2) If Figure 2 As shown, each end uploads the updated face video recovery model M i The parameter θ i , the cloud-side server aggregates these updated parameters as the initial model parameters for the next round of communication, expressed as:

[0089] 3) In the inference phase, the proposed exponential mixing sampling strategy is applied to address the inter-frame identity drift problem that occurs when restoring long videos. Specifically, the latent code of the previous video clip is mixed to enhance the consistency of the identity between frames, which can be expressed as:

[0090]

[0091] where z t (i) represents the hidden layer code of the i-th video clip, the superscript in the formula represents the value range of the frame number, stack(.) represents the vector stacking operation, O is the shared frame length of adjacent video clips, and F is the frame length of each video clip.

[0092] The face video restoration method guided by feedback learning in a device-cloud collaborative environment proposed in this paper outperforms existing methods in various indicators on real-world datasets:

[0093]

[0094] like Figure 3 As shown in the figure, compared with the existing most advanced face restoration model, the present invention can achieve better identity preservation effect in severely degraded scenarios, and the restoration results are significantly improved.

Claims

1. A face video restoration method guided by feedback learning in a device-cloud collaborative environment includes the following steps: 1) Build a face video restoration model based on the Face2Text encoder, face vision encoder, and video restoration base model; 2) During the pre-training phase of the cloud-side face video restoration model, the high-definition identity reference image is parsed into cues in text and visual modalities, and a decoupled cross-attention mechanism is used to guide the model's denoising and restoration process. After pre-training is completed, the obtained model weights are distributed to each end side; 3) During the device-cloud collaborative learning phase, each device-side model performs feedback learning based on identity preservation loss and human preference loss. After each device-side model completes parameter optimization, the optimized model parameters are uploaded to the cloud for aggregation and initialization of parameters required for the next round of communication. 4) In the inference stage of the face video restoration model, multiple inferences are performed on the low-quality video to be restored, and the restored video with the highest reward score is output as the output result.

2. The method for restoring facial videos guided by feedback learning in a device-cloud collaborative environment according to claim 1 is characterized in that: In step 2), the data for the supervised training process includes high-quality videos, corresponding low-quality videos, and identity reference images.

3. The method for restoring facial videos guided by feedback learning in a device-cloud collaborative environment according to claim 1 is characterized in that: The step 2) specifically includes the following steps: 2.1) Generate multimodal cues using high-definition identity reference images, specifically textual and visual cues; in: The text modality hint is obtained by passing the pre-trained Face2Text encoder through the reference image, denoted as c text The Face2Text encoder consists of a facial attribute detector, a large language model, and a CLIP text encoder connected in sequence; The visual modality hint is generated by the pre-trained face visual encoder, denoted as c img ; 2.2) Using a decoupled cross-attention mechanism to perform multimodal cue control on the denoising process of the face video restoration model M; The face video restoration model M is composed of multiple U-Net blocks. Let Z be the input of the U-Net block. Cross-attention is performed on the text and visual modalities respectively and summed up to obtain the decoupled cross-attention output Z′. The process is expressed as: Z′=Attention(Q,Kt,Vt)+λ.Attention(Q,K i ,V i ) Among them, the matrix Q = ZW q , Among them, (Q, K t , V t ) are the query matrix, key matrix and value matrix of the text cross attention operation; K i ,V i is the key matrix and value matrix of the visual cross-attention operation; W q , is the learnable weight; 2.3) Joint High-Resolution Video Reconstruction Loss and identity retention loss Pre-train the cloud-side face video restoration model, and the overall loss is recorded as:

4. The method for restoring facial videos guided by feedback learning in a device-cloud collaborative environment according to claim 2 is characterized in that: In the step 2.3): High-resolution video reconstruction loss The prediction loss for the video diffusion model is expressed as: Among them, c lq Represents a low-resolution video, and controls the denoising process through ControlNet; c text , c img The denoising process is controlled by the decoupled cross-attention mechanism in step 2.2); z is the encoding obtained by inputting the high-resolution video into the Encoder of the variational autoencoder, z t is the result of the forward process of the diffusion model gradually adding noise to the t step, ∈ is the added noise, ∈ θ (.) is the noise predictor of the face video restoration model M, and θ is the corresponding learnable parameter; Loss of identity preservation Expressed as: Among them, cos_sim(.) is used to calculate f ref and The cosine similarity of f ref is the face visual encoding of the reference identity image, denoted as f ref =FaceEnc(ref); is the face visual encoding of the i-th key frame of the generated video, expressed as FaceEnc(.) is a face visual encoder.

5. The method for restoring facial videos guided by feedback learning in a device-cloud collaborative environment according to claim 1 is characterized in that: The step 3) specifically comprises the following steps: 3.1) Identity retention loss in each end-side model and human preference loss For feedback learning, the training loss is as follows: Among them, ζ is a hyperparameter used to adjust the ratio of the two losses; 3.2) Each end uploads the updated face video recovery model M i The parameter θ i , the cloud-side server aggregates these updated parameters as the initial model parameters for the next round of communication. The specific formula is as follows: Where N is the number of devices on the end side.

6. The method for restoring facial videos guided by feedback learning in a device-cloud collaborative environment according to claim 5 is characterized in that: The human preference loss is obtained by the pre-trained reward model gψ, which is specifically expressed as: Among them, x gen Videos generated by the face video restoration model; Reward Model g ψ It is trained on a self-collected human-annotated preference dataset, and its training loss is expressed as: Among them, g ψ (x i , c text , c img ) indicates that the video x i , reference identity text modal prompt c text , visual modal prompt c img When it is input, the reward model g ψ Output reward score.

7. The method for restoring facial videos guided by feedback learning in a device-cloud collaborative environment according to claim 1 is characterized in that: In step 3): in the unsupervised training process of end-cloud collaborative learning, the data used includes low-quality videos and identity reference images.

8. The method for restoring facial videos guided by feedback learning in a device-cloud collaborative environment according to claim 1, characterized in that: In step 4), an exponential mixing sampling strategy is proposed in the inference stage to deal with the inter-frame identity drift problem that occurs when long video is restored. The sampling strategy of the exponential mixing is specifically as follows: The input low-quality video is divided into multiple fixed-length video segments, and the hidden layer encoding of the segmented video segments is mixed to enhance the consistency of identity between frames, which is expressed as: Among them, z t (i) represents the hidden layer code of the i-th video segment at time step t. The superscript in the formula represents the value range of the frame number, stack(.) represents the vector stacking operation, O is the shared frame length of adjacent video segments, and F is the frame length of each video segment.

9. A terminal, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is configured to implement the method for restoring facial videos guided by feedback learning in a terminal-cloud collaborative environment as described in any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by the processor, the method for restoring facial videos guided by feedback learning in an end-cloud collaborative environment as described in any one of claims 1 to 8 is implemented.

Images