A portrait animation generation method and system based on consistency distillation

Abstract

This application belongs to the field of computer vision and video generation, and specifically discloses a method and system for generating portrait animation based on consistency distillation. The method includes: collecting reference image samples and human posture sequence samples as a training set, constructing a pre-trained teacher model based on the training set, and the teacher model is used to generate the target video; initializing the consistency distillation model to obtain a student model, and transferring the knowledge of the teacher model to the student model through the trajectory segmentation consistency distillation method; optimizing the student model through auxiliary loss optimization, dynamic area weighted optimization and facial detail injection to obtain an optimized model; inputting the reference image and human posture sequence to be inferred into the optimized model to obtain a portrait animation video. This application can effectively improve the temporal consistency, coherence and detail fidelity of the generated video.

Description

Technical Field

[0001] The present application belongs to the field of computer vision and video generation, and more specifically, to a method and system for generating portrait animation based on consistency distillation. Background Art

[0002] In recent years, with the advancement of deep learning technology, generative models have made significant progress in the field of portrait animation generation. In particular, generative methods based on diffusion models have attracted considerable attention for their ability to generate high-quality images and videos in complex scenes. Diffusion models demonstrate great potential in video generation by gradually denoising and reconstructing high-resolution content. However, while diffusion models generate high-quality videos, their generation process typically requires hundreds or even thousands of inference steps, resulting in low inference efficiency and difficulty meeting real-time requirements.

[0003] Existing research has proposed several improvements to improve the inference efficiency of diffusion models. For example, downsampling optimization or simplified network structure design can reduce inference steps, or knowledge distillation can be used to optimize a lightweight student model using a teacher model for acceleration. However, these methods often sacrifice the temporal consistency and detail quality of the generated video, resulting in blurry or distorted results in motion regions and facial features. In particular, in motion regions, the generated model is prone to incoherence in the video content. In portrait animation generation, facial details are often the most critical component, but existing methods struggle to achieve both efficient inference and high fidelity.

[0004] Therefore, how to improve the temporal consistency, coherence and facial detail fidelity of portrait animation generated videos, and thus improve the video quality, is a technical problem that needs to be solved urgently. Summary of the Invention

[0005] In response to the shortcomings of the existing technology, the purpose of this application is to provide a portrait animation generation method and system based on consistency distillation, which aims to solve the problems of poor temporal consistency, video coherence and detail quality of videos in the existing technology.

[0006] To achieve the above objectives, in a first aspect, the present application provides a method for generating portrait animation based on consistency distillation, comprising:

[0007] Collect reference image samples and human posture sequence samples as a training set, build a pre-trained teacher model based on the training set, and use the teacher model to generate a target video;

[0008] Initialize the consistency distillation model to obtain the student model, and transfer the knowledge of the teacher model to the student model through the trajectory segmentation consistency distillation method;

[0009] Optimizing the student model through auxiliary loss optimization, dynamic region weighted optimization, and facial detail injection to obtain an optimized model;

[0010] The reference image to be inferred and the human posture sequence are input into the optimization model to obtain a portrait animation video.

[0011] Optionally, transferring the knowledge of the teacher model to the student model through the trajectory segment consistency distillation method includes:

[0012] Divide the probability trajectory into K segments and set the boundary time step of each segment;

[0013] The objective function is used to optimize self-consistency within each trajectory, and the output results of the teacher model and the student model are compared and optimized so that the student model can continuously imitate the target video generated by the teacher model.

[0014] Optionally, the objective function is:

[0015]

[0016] in, and are the boundary time steps of two different segments, represents the boundary time step, , is the student model parameter The exponential moving average of For the student model at the boundary time step Input, For the teacher model based on trajectory points The predicted output.

[0017] Optionally, the specific process of the auxiliary loss optimization includes:

[0018] Obtain the true latent variables of the target video generated by the teacher model and the latent variables to be optimized for the predicted video output by the student model;

[0019] The latent variable to be optimized is compared with the actual latent variable to obtain the latent variable difference, and the latent variable difference is optimized using the optimization auxiliary loss function to improve the temporal consistency and spatial details of the predicted video.

[0020] Optionally, the specific process of the dynamic area weighted optimization includes:

[0021] Determining a target pixel value at a target position of a current video frame, a first pixel value at the target position of a first video frame, and a second pixel value at the target position of a second video frame; wherein the first video frame is a frame preceding the current video frame and the second video frame is a frame following the current video frame;

[0022] Constructing a dynamic region set between video frames of a predicted video output by a student model according to the target pixel value, the first pixel value, the second pixel value, and a preset threshold;

[0023] The dynamic region set and the weighted balance factor are used to obtain the dynamic region loss, and the overall loss between the predicted video and the target video is obtained. The dynamic region loss and the overall loss are dynamically weighted optimized for the predicted video through a dynamic region optimization loss function.

[0024] Optionally, the dynamic region set is defined as:

[0025]

[0026] in, is the preset threshold for dynamic detection, is the pixel coordinate, is the target pixel value of the current video frame, is the second pixel value, is the first pixel value.

[0027] Optionally, the specific process of injecting facial details includes:

[0028] extracting a facial region of the reference image sample;

[0029] Encoding the facial region using a variational autoencoder to obtain a high-fidelity facial feature representation;

[0030] Concatenating and fusing the facial feature representation with the global features of the CLIP image encoder to obtain a fused facial feature;

[0031] The fused facial features are input into the student model for optimization using a cross-attention mechanism to enhance the fidelity of facial details.

[0032] Optionally, the student model is optimized based on a total loss function, where the total loss function is determined based on optimizing the auxiliary loss function and the dynamic region optimization loss function, as shown in the following formula:

[0033]

[0034]

[0035]

[0036] in, Optimize the loss function for dynamic regions, To optimize the auxiliary loss function;

[0037] is the overall loss between the predicted video and the target video, is the weighted balancing factor, is the dynamic area loss;

[0038] is the video latent variable, is the latent variable to be optimized; Weight factor optimized for the auxiliary loss.

[0039] In a second aspect, the present application further provides a portrait animation generation system based on consistency distillation, comprising:

[0040] a teacher model determination unit, configured to collect reference image samples and human posture sequence samples as a training set, and construct a pre-trained teacher model based on the training set, wherein the teacher model is used to generate a target video;

[0041] A trajectory segmentation consistency distillation unit is used to initialize the consistency distillation model to obtain a student model, and transfer the knowledge of the teacher model to the student model through the trajectory segmentation consistency distillation method;

[0042] an optimization unit, configured to optimize the student model by auxiliary loss optimization, dynamic region weighted optimization, and facial detail injection to obtain an optimized model;

[0043] The inference generation unit is used to input the reference image to be inferred and the human posture sequence into the optimization model to obtain a portrait animation video.

[0044] In a third aspect, the present application provides an electronic device comprising: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method described in the first aspect or any possible implementation of the first aspect.

[0045] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect.

[0046] In a fifth aspect, the present application provides a computer program product, which, when executed on a processor, enables the processor to execute the method described in the first aspect or any possible implementation of the first aspect.

[0047] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0048] In general, the above technical solutions conceived by this application have the following beneficial effects compared with the existing technologies:

[0049] (1) This application can effectively improve the temporal consistency and spatial detail quality of the generated video through trajectory segment consistency distillation and auxiliary supervision mechanism, and ensure the smooth transition of the generated portrait animation on the timeline; by utilizing dynamic area weighted optimization, the model focuses more on the motion area in the video, thereby enhancing the performance of dynamic effects; by injecting facial details, the authenticity of facial features is ensured, and when generating dynamic videos, the smoothness of character movements can be better captured and maintained, thereby improving the temporal consistency, coherence and facial detail fidelity of the video, thereby improving video quality.

[0050] (2) This application optimizes the reasoning steps in the generation process by introducing consistency distillation technology. The student model reduces the amount of calculation and significantly reduces the reasoning time through the consistency distillation method, making the process of generating high-quality portrait animation more efficient and able to meet the needs of fast reasoning and real-time applications, especially suitable for practical scenarios with limited computing resources.

[0051] (3) This application can detect and focus on optimizing the moving areas in the video through motion area optimization, effectively reducing the motion blur and detail distortion problems that are easily caused by traditional methods, ensuring the clarity and detail fidelity of the moving areas, and improving the overall visual effect of the video.

[0052] (4) This application uses a variational autoencoder to extract facial features from a reference image and injects a consistent distillation model with a cross-attention mechanism, significantly improving the fidelity and realism of facial features in the generated video. Compared with existing technologies, facial details are more accurate, avoiding the problem of facial expression and detail distortion in existing methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is one of the flow charts of the method for generating portrait animation based on consistency distillation provided in an embodiment of the present application;

[0054] Figure 2 This is the second flow chart of the method for generating portrait animation based on consistency distillation provided in an embodiment of the present application;

[0055] Figure 3 This is the third flow chart of the method for generating portrait animation based on consistency distillation provided in an embodiment of the present application;

[0056] Figure 4 Schematic diagram of the structure of a portrait animation generation system based on consistency distillation provided in an embodiment of the present application;

[0057] Figure 5 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0059] The term "and / or" as used herein describes an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. The symbol " / " as used herein indicates that the related objects are in an "or" relationship, for example, A / B means either A or B.

[0060] The terms "first" and "second" in this specification and claims are used to distinguish different objects rather than to describe a specific order of objects. For example, "first response message" and "second response message" are used to distinguish different response messages rather than to describe a specific order of response messages.

[0061] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0062] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more, for example, multiple processing units means two or more processing units, etc.; multiple elements means two or more elements, etc.

[0063] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application.

[0064] Reference Figure 1 , this application provides a method for generating portrait animation based on consistency distillation, comprising:

[0065] S101. Collect reference image samples and human posture sequence samples as a training set, and construct a pre-trained teacher model based on the training set, wherein the teacher model is used to generate a target video;

[0066] S102. Initialize the consistency distillation model to obtain a student model, and transfer the knowledge of the teacher model to the student model through the trajectory segmentation consistency distillation method;

[0067] S103. Optimizing the student model by auxiliary loss optimization, dynamic region weighted optimization, and facial detail injection to obtain an optimized model;

[0068] S104. Input the reference image to be inferred and the human posture sequence into the optimization model to obtain a portrait animation video.

[0069] Specifically, the embodiment of the present application first collects reference image samples and human pose sequence samples through S101 to form a training set. This data is used to train and optimize the model to ensure that the generated video accurately reflects the target content. Based on the collected training set, a pre-trained teacher model is constructed. The teacher model is responsible for generating the target video. The teacher model has good generation capabilities and stability and is used to guide the learning of the student model.

[0070] Next, the consistency distillation model is initialized in step S102 to produce a student model. The student model is a lightweight model designed to mimic the generative capabilities of the teacher model. Through trajectory segmentation consistency distillation, the knowledge of the teacher model is transferred to the student model. During this process, the student model learns by imitating the results generated by the teacher model at different time periods, enhancing its generative capabilities.

[0071] Furthermore, in step S103, a series of optimizations are performed on the student model, including the following aspects:

[0072] Auxiliary loss optimization: Auxiliary loss is introduced to improve the performance of the model during the generation process.

[0073] Weighted Dynamic Area Optimization: Pays special attention to dynamic areas, enhancing their detail and ensuring smooth and dynamic animations.

[0074] Facial detail injection: Extract and enhance facial details in generated videos to improve the realism of human expressions and features.

[0075] After the above optimization process, an optimized student model is finally obtained, which can better generate high-quality portrait animation.

[0076] Specifically, lightweight auxiliary supervision: During the distillation process, the feature distribution of the video latent variable is obtained through the lightweight auxiliary head module and aligned with the real video latent variable; the auxiliary supervision signal is used to enhance the temporal consistency and spatial detail quality of the generated video;

[0077] Dynamic Area Focused Optimization: Detects dynamic areas in the input video based on inter-frame differences and performs weighted optimization on these areas to reduce blur and distortion in moving areas.

[0078] Facial fidelity feature injection: Extract facial features from the reference image, encode them through a variational autoencoder, and fuse them with global features; use a cross-attention mechanism to model facial features and inject them into a consistency distillation model to improve the realism of facial details in the generated video.

[0079] Finally, in step S104, the reference image to be inferred and the human pose sequence are input into the optimized student model. After processing by the optimized model, the portrait animation video is output, ensuring that it achieves good results in terms of quality and dynamic performance.

[0080] Optionally, transferring the knowledge of the teacher model to the student model through the trajectory segment consistency distillation method includes:

[0081] Divide the probability trajectory into K segments and set the boundary time step of each segment;

[0082] The objective function is used to optimize self-consistency within each trajectory, and the output results of the teacher model and the student model are compared and optimized so that the student model can continuously imitate the target video generated by the teacher model.

[0083] Optionally, the objective function is:

[0084]

[0085] in, and are the boundary time steps of two different segments, represents the boundary time step, , is the student model parameter The exponential moving average of For the student model at the boundary time step Input, For the teacher model based on trajectory points The predicted output.

[0086] Specifically, in this application, in order to optimize the temporal consistency in the generation process, the probability flow trajectory is divided into segments, and set the boundary time step of each segment to , which reduces the learning difficulty by optimizing the self-consistency within each trajectory.

[0087] Optionally, the specific process of the auxiliary loss optimization includes:

[0088] Obtain the true latent variables of the target video generated by the teacher model and the latent variables to be optimized for the predicted video output by the student model;

[0089] The latent variable to be optimized is compared with the actual latent variable to obtain the latent variable difference, and the latent variable difference is optimized using the optimization auxiliary loss function to improve the temporal consistency and spatial details of the predicted video.

[0090] During the distillation process, a lightweight auxiliary head module is introduced to generate the video latent variables. Make predictions and compare them with the real video latent variables Alignment improves the temporal consistency and spatial details of the generated video by optimizing the following auxiliary loss function:

[0091]

[0092] in, is the video latent variable, is the latent variable to be optimized.

[0093] Optionally, the specific process of the dynamic area weighted optimization includes:

[0094] Determining a target pixel value at a target position of a current video frame, a first pixel value at the target position of a first video frame, and a second pixel value at the target position of a second video frame; wherein the first video frame is a frame preceding the current video frame and the second video frame is a frame following the current video frame;

[0095] Constructing a dynamic region set between video frames of a predicted video output by a student model according to the target pixel value, the first pixel value, the second pixel value, and a preset threshold;

[0096] The dynamic region set and the weighted balance factor are used to obtain the dynamic region loss, and the overall loss between the predicted video and the target video is obtained. The dynamic region loss and the overall loss are dynamically weighted optimized for the predicted video through a dynamic region optimization loss function.

[0097] Specifically, the embodiment of the present application detects dynamic areas by calculating pixel differences between input video frames, and the dynamic area set is defined as:

[0098]

[0099] in, is the preset threshold for dynamic detection, is the pixel coordinate, is the target pixel value of the current video frame, is the second pixel value, is the first pixel value.

[0100] In the dynamic region, the loss is given a higher weight, and the optimization objective function is:

[0101]

[0102] in, is the overall loss between the predicted video and the target video, is the weighted balancing factor, is the dynamic area loss;

[0103] Optionally, the specific process of injecting facial details includes:

[0104] extracting a facial region of the reference image sample;

[0105] Encoding the facial region using a variational autoencoder to obtain a high-fidelity facial feature representation;

[0106] Concatenating and fusing the facial feature representation with the global features of the CLIP image encoder to obtain a fused facial feature;

[0107] The fused facial features are input into the student model for optimization using a cross-attention mechanism to enhance the fidelity of facial details.

[0108] Specifically, the embodiment of the present application extracts the facial area of ​​the reference image, encodes it through a variational autoencoder, generates a high-fidelity facial feature representation, splices it with the global features of the CLIP image encoder, and uses a cross-attention mechanism to inject the facial features into the consistency distillation model to enhance the realism and consistency of facial details.

[0109] Optionally, the student model is optimized based on a total loss function, where the total loss function is determined based on optimizing the auxiliary loss function and the dynamic region optimization loss function, as shown in the following formula:

[0110]

[0111]

[0112]

[0113] in, is the overall loss between the predicted video and the target video, is the weighted balancing factor, is the dynamic area loss;

[0114] is the video latent variable, is the latent variable to be optimized; λ2 is the weight factor for auxiliary loss optimization.

[0115] Reference Figure 2 The portrait animation generation method based on consistency distillation provided in the embodiment of the present application specifically includes the following steps:

[0116] Determine the consistency distillation model: pre-train a video diffusion model as the teacher model, which generates target videos based on reference images and human pose sequences. Initialize the consistency distillation model as the student model, which is optimized using the consistency distillation method with the goal of reducing the number of inference steps.

[0117] (2) Trajectory segment consistency distillation: Divide the probability stream trajectory into multiple time periods and optimize the different feature stages of each segment separately; by optimizing the consistency of adjacent trajectory points in the segmented trajectory, the optimization difficulty is reduced and the error accumulation is reduced;

[0118] (3) Lightweight auxiliary supervision: During the distillation process, the feature distribution of the video latent variable is obtained through the lightweight auxiliary head module and aligned with the real video latent variable; the auxiliary supervision signal is used to enhance the temporal consistency and spatial detail quality of the generated video;

[0119] (4) Dynamic Region Focused Optimization: Detects dynamic regions in the input video based on inter-frame differences and performs weighted optimization on these regions to reduce blur and distortion in moving areas;

[0120] (5) Facial fidelity feature injection: extract facial features from the reference image, encode them through a variational autoencoder, and then fuse them with global features; use a cross-attention mechanism to model facial features and inject them into a consistency distillation model to improve the realism of facial details in the generated video;

[0121] (6) Reasoning and Generation: Input reference images and human pose sequences, use the optimized student model, and generate high-quality and time-consistent portrait animation videos through 2-4 steps of reasoning.

[0122] Reference Figure 3 This embodiment provides a specific method for generating portrait animation based on consistency distillation, including the following steps:

[0123] (1) Data collection and model initialization;

[0124] To complete the task of portrait animation generation, it is first necessary to collect appropriate training data, mainly including reference images and human pose sequences. This data will serve as input for the teacher model and the student model. The reference image is static but contains sufficient facial and background information, which will provide visual details for the final generated video. The human pose sequence is obtained through key point detection technology, which represents the position information of various parts of the human body. This data is used to guide the character movements in animation generation. In this step, a pre-trained video diffusion model is used as the teacher model. The teacher model already has the ability to generate high-quality videos, so it can generate target videos based on the reference image and human pose sequence. This provides a standard generation target for subsequent student model learning.

[0125] (2) Constructing a consistent distilled portrait animation generation network;

[0126] In this phase, we build a portrait animation generation network based on consistency distillation. The network architecture includes the following key modules:

[0127] Teacher Model: A pre-trained video diffusion model used to generate target videos. The teacher model does not need to be optimized during the generation process; it provides known high-quality video generation capabilities.

[0128] Student model: Compared to the teacher model, the student model is more lightweight and aims to quickly learn and efficiently generate videos through consistent distillation. The student model is optimized to mimic the teacher model's video generation process.

[0129] Lightweight Auxiliary Head Module: This module helps the student model capture the characteristic distribution of the video latent variables during the generation process and aligns them with the latent variables of the real video. By comparing the differences between the generated latent variables and the real latent variables, the generated video is optimized in terms of temporal consistency and spatial detail.

[0130] Dynamic Area Optimization Module: The goal of this module is to focus on optimizing the moving areas in the video, reduce motion blur by weighted processing of dynamic areas, and ensure the clarity and consistency of details in the moving areas.

[0131] Facial Detail Preservation Module: This module extracts facial features from the reference image and encodes them using a variational autoencoder (VAE). These facial features are injected into the student model via a cross-attention mechanism, enhancing the fidelity of facial details in the generated video and ensuring the realism of facial details.

[0132] (3) Knowledge transfer and generation optimization;

[0133] In this step, the knowledge of the teacher model is transferred to the student model through the consistency distillation method. The specific steps are as follows:

[0134] (3-1) Trajectory Segment-wise Consistency Distillation: The input video trajectory is divided into multiple time segments, and the model performs self-consistency optimization within each segment. By optimizing the consistency of trajectory points within each segment, the learning difficulty is reduced and error accumulation is minimized. This segment-wise distillation method effectively avoids the optimization difficulties over long time spans.

[0135] (3-2) Auxiliary Loss Optimization: By introducing an auxiliary head module, the student model can predict the latent variables of the video and compare them with the latent variables of the real video. By optimizing the auxiliary loss function, the temporal consistency and spatial details of the video are further improved. This process ensures that the student model not only imitates the video content generated by the teacher model, but also produces more stable and natural videos.

[0136] (3-3) Weighted Optimization of Dynamic Regions: During video generation, the model calculates pixel differences between video frames to identify dynamic regions. These dynamic regions typically contain parts that move or change rapidly. By weighting and optimizing these dynamic regions, motion blur and distortion are reduced, ensuring the clarity and realism of the generated video in these dynamic regions.

[0137] (3-4) Facial detail injection: Facial features in the reference image are encoded using a variational autoencoder and fused with global features. These facial features are then injected into the student model using a cross-attention mechanism to enhance the fidelity of facial details and ensure that the generated video can present high-quality facial details.

[0138] (4) Rapid inference to generate portrait animation;

[0139] During the inference phase, the model takes as input a reference image and a sequence of human poses and uses an optimized student model for rapid inference. Compared to traditional generative methods, the student model, optimized through consistency distillation, significantly accelerates inference, generating high-quality, temporally consistent dynamic videos in just 2-4 steps. The resulting videos are not only more realistic in spatial detail but also maintain temporal consistency, resulting in smooth animation.

[0140] Reference Figure 4 , this application also provides a portrait animation generation system based on consistency distillation, including:

[0141] a teacher model determination unit 410 for collecting reference image samples and human posture sequence samples as a training set, and constructing a pre-trained teacher model based on the training set, wherein the teacher model is used to generate a target video;

[0142] A trajectory segment-wise consistency distillation unit 420 is configured to initialize a consistency distillation model to obtain a student model, and transfer the knowledge of the teacher model to the student model through a trajectory segment-wise consistency distillation method;

[0143] an optimization unit 430, configured to optimize the student model by auxiliary loss optimization, dynamic region weighted optimization, and facial detail injection to obtain an optimized model;

[0144] The inference generation unit 440 is used to input the reference image to be inferred and the human posture sequence into the optimization model to obtain a portrait animation video.

[0145] Optionally, transferring the knowledge of the teacher model to the student model through the trajectory segment consistency distillation method includes:

[0146] Divide the probability trajectory into K segments and set the boundary time step of each segment;

[0147] The objective function is used to optimize self-consistency within each trajectory, and the output results of the teacher model and the student model are compared and optimized so that the student model can continuously imitate the target video generated by the teacher model.

[0148] Optionally, the objective function is:

[0149]

[0150] in, and are the boundary time steps of two different segments, represents the boundary time step, , is the student model parameter The exponential moving average of For the student model at the boundary time step Input, For the teacher model based on trajectory points The predicted output.

[0151] Optionally, the specific process of the auxiliary loss optimization includes:

[0152] Obtain the true latent variables of the target video generated by the teacher model and the latent variables to be optimized for the predicted video output by the student model;

[0153] The latent variable to be optimized is compared with the actual latent variable to obtain the latent variable difference, and the latent variable difference is optimized using the optimization auxiliary loss function to improve the temporal consistency and spatial details of the predicted video.

[0154] Optionally, the specific process of the dynamic area weighted optimization includes:

[0155] Determining a target pixel value at a target position of a current video frame, a first pixel value at the target position of a first video frame, and a second pixel value at the target position of a second video frame; wherein the first video frame is a frame preceding the current video frame and the second video frame is a frame following the current video frame;

[0156] Constructing a dynamic region set between video frames of a predicted video output by a student model according to the target pixel value, the first pixel value, the second pixel value, and a preset threshold;

[0157] The dynamic region set and the weighted balance factor are used to obtain the dynamic region loss, and the overall loss between the predicted video and the target video is obtained. The dynamic region loss and the overall loss are dynamically weighted optimized for the predicted video through a dynamic region optimization loss function.

[0158] Optionally, the dynamic region set is defined as:

[0159]

[0160] in, is the preset threshold for dynamic detection, is the pixel coordinate, is the target pixel value of the current video frame, is the second pixel value, is the first pixel value.

[0161] Optionally, the specific process of injecting facial details includes:

[0162] extracting a facial region of the reference image sample;

[0163] Encoding the facial region using a variational autoencoder to obtain a high-fidelity facial feature representation;

[0164] Concatenating and fusing the facial feature representation with the global features of the CLIP image encoder to obtain a fused facial feature;

[0165] The fused facial features are input into the student model for optimization using a cross-attention mechanism to enhance the fidelity of facial details.

[0166] Optionally, the student model is optimized based on a total loss function, where the total loss function is determined based on optimizing the auxiliary loss function and the dynamic region optimization loss function, as shown in the following formula:

[0167]

[0168]

[0169]

[0170] in, Optimize the loss function for dynamic regions, To optimize the auxiliary loss function;

[0171] is the overall loss between the predicted video and the target video, is the weighted balancing factor, is the dynamic area loss;

[0172] is the video latent variable, is the latent variable to be optimized; λ2 is the weight factor for auxiliary loss optimization.

[0173] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method and will not be repeated here.

[0174] Reference Figure 5 Based on the methods in the above embodiments, an embodiment of the present application provides an electronic device, which may include: a processor (Processor) 510, a communication interface (Communications Interface) 520, a memory (Memory) 530, and a communication bus 540. The processor 510, the communication interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 may call logic instructions in the memory 530 to execute the methods in the above embodiments.

[0175] In addition, the logic instructions in the aforementioned memory 530 can be implemented in the form of a software functional unit and, when sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application.

[0176] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method in the above embodiment.

[0177] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method in the above embodiment.

[0178] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0179] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and storage medium can be located in an ASIC.

[0180] The above embodiments can be implemented in whole or in part using software, hardware, firmware, or any combination thereof. When implemented using software, they can be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions. When loaded and executed on a computer, the computer program instructions fully or partially produce the processes or functions described in the embodiments of this application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted via the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic media (e.g., floppy disk, hard disk, tape), optical media (e.g., DVD), or semiconductor media (e.g., solid-state drive (SSD)).

[0181] It will be understood that the various numerical numbers involved in the embodiments of the present application are merely distinctions for the convenience of description and are not intended to limit the scope of the embodiments of the present application.

[0182] It is easy for those skilled in the art to understand that the above is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A portrait animation generation method based on consistency distillation, characterized in that: include: Collect reference image samples and human posture sequence samples as a training set, build a pre-trained teacher model based on the training set, and use the teacher model to generate a target video; Initialize the consistency distillation model to obtain the student model, and transfer the knowledge of the teacher model to the student model through the trajectory segmentation consistency distillation method; Optimizing the student model through auxiliary loss optimization, dynamic region weighted optimization, and facial detail injection to obtain an optimized model; Inputting the reference image to be inferred and the human posture sequence into the optimization model to obtain a portrait animation video; The method of transferring the knowledge of the teacher model to the student model through the trajectory segment consistency distillation method includes: Divide the probability flow trajectory into K segments and set the boundary time step of each segment; Using the objective function to optimize self-consistency within each trajectory, the output of the teacher model and the student model are compared and optimized so that the student model can continuously imitate the target video generated by the teacher model. The specific process of the auxiliary loss optimization includes: Obtain the true latent variables of the target video generated by the teacher model and the latent variables to be optimized for the predicted video output by the student model; The latent variable to be optimized is compared with the actual latent variable to obtain the latent variable difference, and the latent variable difference is optimized using the optimization auxiliary loss function to improve the temporal consistency and spatial details of the predicted video.

2. The method for generating portrait animation based on consistency distillation according to claim 1, characterized in that: The objective function is: in, and are the boundary time steps of two different segments, represents the boundary time step, , is the student model parameter The exponential moving average of For the student model at the boundary time step Input, For the teacher model based on trajectory points The predicted output.

3. The method for generating portrait animation based on consistency distillation according to claim 1, characterized in that: The specific process of the dynamic regional weighted optimization includes: Determining a target pixel value at a target position of a current video frame, a first pixel value at the target position of a first video frame, and a second pixel value at the target position of a second video frame; wherein the first video frame is a frame preceding the current video frame and the second video frame is a frame following the current video frame; Constructing a dynamic region set between video frames of a predicted video output by a student model according to the target pixel value, the first pixel value, the second pixel value, and a preset threshold; The dynamic region set and the weighted balance factor are used to obtain the dynamic region loss, and the overall loss between the predicted video and the target video is obtained. The dynamic region loss and the overall loss are dynamically weighted optimized for the predicted video through a dynamic region optimization loss function.

4. The method for generating portrait animation based on consistency distillation according to claim 3, characterized in that: The dynamic region set is defined as: in, is the preset threshold for dynamic detection, is the pixel coordinate, is the target pixel value of the current video frame, is the second pixel value, is the first pixel value.

5. The method for generating portrait animation based on consistency distillation according to claim 1, characterized in that: The specific process of injecting facial details includes: extracting a facial region of the reference image sample; Encoding the facial region using a variational autoencoder to obtain a high-fidelity facial feature representation; Concatenating and fusing the facial feature representation with the global features of the CLIP image encoder to obtain a fused facial feature; The fused facial features are input into the student model for optimization using a cross-attention mechanism to enhance the fidelity of facial details.

6. The method for generating portrait animation based on consistency distillation according to claim 4, characterized in that: The student model is optimized based on the total loss function, which is determined based on the optimization auxiliary loss function and the dynamic area optimization loss function, as shown in the following formula: in, Optimize the loss function for dynamic regions, To optimize the auxiliary loss function; is the overall loss between the predicted video and the target video, is the weighted balancing factor, is the dynamic area loss; is the video latent variable, is the latent variable to be optimized; Weight factor optimized for the auxiliary loss.

7. A portrait animation generation system based on consistency distillation, characterized by: include: a teacher model determination unit, configured to collect reference image samples and human posture sequence samples as a training set, and construct a pre-trained teacher model based on the training set, wherein the teacher model is used to generate a target video; A trajectory segmentation consistency distillation unit is used to initialize the consistency distillation model to obtain a student model, and transfer the knowledge of the teacher model to the student model through the trajectory segmentation consistency distillation method; an optimization unit, configured to optimize the student model by auxiliary loss optimization, dynamic region weighted optimization, and facial detail injection to obtain an optimized model; An inference generation unit, configured to input a reference image to be inferred and a human posture sequence into the optimization model to obtain a portrait animation video; The method of transferring the knowledge of the teacher model to the student model through the trajectory segment consistency distillation method includes: Divide the probability trajectory into K segments and set the boundary time step of each segment; Using the objective function to optimize self-consistency within each trajectory, the output of the teacher model and the student model are compared and optimized so that the student model can continuously imitate the target video generated by the teacher model. The specific process of the auxiliary loss optimization includes: Obtain the true latent variables of the target video generated by the teacher model and the latent variables to be optimized for the predicted video output by the student model; The latent variable to be optimized is compared with the actual latent variable to obtain the latent variable difference, and the latent variable difference is optimized using the optimization auxiliary loss function to improve the temporal consistency and spatial details of the predicted video.

8. An electronic device, characterized in that: include: at least one memory for storing a computer program; At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method according to any one of claims 1 to 6.

Images