Portrait animation generation method and system based on consistent distillation

Abstract

The invention belongs to the field of computer vision and video generation, and particularly discloses a portrait animation generation method and system based on consistent distillation. The method comprises the steps that a reference image sample and a human body posture sequence sample are collected to serve as a training set, a pre-trained teacher model is constructed based on the training set, and the teacher model is used for generating a target video; initializing a consistency distillation model to obtain a student model, and transmitting knowledge of the teacher model to the student model through a track segmentation consistency distillation method; optimizing the student model through auxiliary loss optimization, dynamic region weighted optimization and facial detail injection to obtain an optimized model; and inputting a reference image to be reasoned and the human body posture sequence into the optimization model to obtain a portrait animation video. According to the invention, the time sequence consistency, coherence and detail fidelity of the generated video can be effectively improved.

Description

Technical Field

[0001] The present application belongs to the field of computer vision and video generation, and more specifically, to a method and system for generating portrait animation based on consistency distillation. Background Art

[0002] In recent years, with the development of deep learning technology, generative models have made significant progress in the field of portrait animation generation. In particular, the generation method based on the diffusion model has attracted much attention due to its ability to generate high-quality images and videos in complex scenes. The diffusion model shows great potential in the field of video generation by gradually denoising and reconstructing high-resolution content. However, although the video generated by the diffusion model is of high quality, its generation process usually requires hundreds or even thousands of steps of reasoning, resulting in low reasoning efficiency and difficulty in meeting real-time requirements.

[0003] Existing studies have proposed some improvement schemes to improve the reasoning efficiency of diffusion models. For example, reducing the reasoning steps by downsampling optimization or designing a simplified network structure; or using knowledge distillation to optimize the lightweight student model using the teacher model to achieve acceleration. However, these methods often sacrifice the temporal consistency and detail quality of the generated video, making the generated results appear blurry or distorted in motion areas and facial features. Especially in motion areas, the generated model is prone to incoherence of video content; and in portrait animation generation, facial details are often the most critical part, but existing methods have difficulty in balancing detail fidelity while achieving efficient reasoning.

[0004] Therefore, how to improve the timing consistency, coherence and facial detail fidelity of portrait animation generated videos, and thus improve the video quality, is a technical problem that needs to be solved urgently. Summary of the invention

[0005] In view of the defects of the prior art, the purpose of this application is to provide a portrait animation generation method and system based on consistency distillation, which aims to solve the problems of poor timing consistency, video coherence and detail quality of videos in the prior art.

[0006] To achieve the above objectives, in a first aspect, the present application provides a method for generating a portrait animation based on consistency distillation, comprising: Collect reference image samples and human posture sequence samples as a training set, build a pre-trained teacher model based on the training set, and use the teacher model to generate a target video; Initialize the consistency distillation model to obtain the student model, and transfer the knowledge of the teacher model to the student model through the trajectory segmentation consistency distillation method; The student model is optimized by auxiliary loss optimization, dynamic region weighted optimization and facial detail injection to obtain an optimized model; The reference image to be inferred and the human posture sequence are input into the optimization model to obtain a portrait animation video.

[0007] Optionally, transferring the knowledge of the teacher model to the student model by using the trajectory segment consistency distillation method includes: Divide the probability trajectory into K segments and set the boundary time step of each segment; The objective function is used to optimize self-consistency within each trajectory, and the output results of the teacher model and the student model are compared and optimized so that the student model can continuously imitate the target video generated by the teacher model.

[0008] Optionally, the objective function is:

[0009] in, and are the boundary time steps of two different segments, represents the boundary time step, , is the student model parameter The exponential moving average of For the student model at the boundary time step Input, Based on trajectory points for the teacher model The predicted output.

[0010] Optionally, the specific process of the auxiliary loss optimization includes: Obtain the true latent variables of the target video generated by the teacher model and the latent variables to be optimized of the predicted video output by the student model; The latent variable to be optimized is compared with the real latent variable to obtain the latent variable difference, and the latent variable difference is optimized by optimizing the auxiliary loss function to improve the temporal consistency and spatial details of the predicted video.

[0011] Optionally, the specific process of the dynamic area weighted optimization includes: Determine a target pixel value of a current video frame at a target position, a first pixel value of a first video frame at the target position, and a second pixel value of a second video frame at the target position; the first video frame is a previous frame of the current video frame, and the second video frame is a subsequent frame of the current video frame; Constructing a dynamic region set between video frames of a predicted video output by a student model according to the target pixel value, the first pixel value, the second pixel value and a preset threshold; The dynamic region set and the weighted balance factor are used to obtain the dynamic region loss, and the overall loss between the predicted video and the target video is obtained. The dynamic region loss and the overall loss are dynamically weighted optimized for the predicted video through a dynamic region optimization loss function.

[0012] Optionally, the dynamic region set is defined as:

[0013] in, is the preset threshold for dynamic detection, is the pixel coordinate, is the target pixel value of the current video frame, is the second pixel value, is the first pixel value.

[0014] Optionally, the specific process of injecting facial details includes: Extracting a facial region of the reference image sample; Encoding the facial region using a variational autoencoder to obtain a high-fidelity facial feature representation; Concatenating and fusing the facial feature representation with the global features of the CLIP image encoder to obtain a fused facial feature; The fused facial features are input into the student model for optimization using a cross-attention mechanism to enhance the fidelity of facial details.

[0015] Optionally, the student model is optimized based on a total loss function, and the total loss function is determined based on optimizing an auxiliary loss function and a dynamic region optimization loss function, as shown in the following formula:

[0016]

[0017]

[0018] in, Optimize the loss function for dynamic regions, To optimize the auxiliary loss function; is the overall loss between the predicted video and the target video, is the weighted balancing factor, is the dynamic area loss; is the video latent variable, is the latent variable to be optimized; Weight factor optimized for the auxiliary loss.

[0019] In a second aspect, the present application also provides a portrait animation generation system based on consistency distillation, comprising: A teacher model determination unit, used to collect reference image samples and human posture sequence samples as a training set, and to construct a pre-trained teacher model based on the training set, wherein the teacher model is used to generate a target video; A trajectory segmentation consistency distillation unit, used to initialize the consistency distillation model to obtain a student model, and transfer the knowledge of the teacher model to the student model through the trajectory segmentation consistency distillation method; An optimization unit, used for optimizing the student model through auxiliary loss optimization, dynamic area weighted optimization and facial detail injection to obtain an optimized model; The inference generation unit is used to input the reference image to be inferred and the human posture sequence into the optimization model to obtain a portrait animation video.

[0020] In a third aspect, the present application provides an electronic device comprising: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method described in the first aspect or any possible implementation of the first aspect.

[0021] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect.

[0022] In a fifth aspect, the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect.

[0023] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0024] In general, the above technical solutions conceived by this application have the following beneficial effects compared with the prior art: (1) This application can effectively improve the temporal consistency and spatial detail quality of the generated video, the temporal consistency and coherence, and ensure the smooth transition of the generated portrait animation on the timeline through trajectory segment consistency distillation and auxiliary supervision mechanism; by utilizing dynamic area weighted optimization, the model focuses more on the motion area in the video, thereby enhancing the performance of dynamic effects; by injecting facial details, the authenticity of facial features is ensured, and when generating dynamic videos, the smoothness of character movements can be better captured and maintained, thereby improving the temporal consistency, coherence and facial detail fidelity of the video, thereby improving video quality.

[0025] (2) This application optimizes the reasoning steps in the generation process by introducing consistency distillation technology. The student model reduces the amount of calculation through the consistency distillation method and significantly reduces the reasoning time, making the process of generating high-quality portrait animation more efficient and able to meet the needs of fast reasoning and real-time applications, especially suitable for practical scenarios with limited computing resources.

[0026] (3) This application can detect and focus on optimizing the moving areas in the video through motion area focused optimization, effectively reducing the motion blur and detail distortion problems that are easily caused by traditional methods, ensuring the clarity and detail fidelity of the moving areas, and improving the overall visual effect of the video. (4) This application extracts facial features from the reference image through a variational autoencoder and injects a consistency distillation model with a cross-attention mechanism, which significantly improves the fidelity and realism of facial features in the generated video. Compared with the existing technology, facial details are more accurate, avoiding the problem of facial expression and detail distortion in the existing method. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is one of the flow charts of the method for generating a portrait animation based on consistency distillation provided in an embodiment of the present application; Figure 2 This is the second flow chart of the method for generating a portrait animation based on consistency distillation provided in an embodiment of the present application; Figure 3 This is the third flow chart of the method for generating a portrait animation based on consistency distillation provided in an embodiment of the present application; Figure 4 is a structural schematic diagram of a portrait animation generation system based on consistency distillation provided in an embodiment of the present application; Figure 5 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0029] The term "and / or" in this article is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this article indicates that the associated objects are in an or relationship, for example, A / B means A or B.

[0030] The terms "first" and "second" in the specification and claims herein are used to distinguish different objects rather than to describe a specific order of the objects. For example, a first response message and a second response message are used to distinguish different response messages rather than to describe a specific order of the response messages.

[0031] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0032] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. For example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0033] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application.

[0034] Reference Figure 1 , the present application provides a method for generating portrait animation based on consistency distillation, comprising: S101. Collect reference image samples and human posture sequence samples as a training set, and construct a pre-trained teacher model based on the training set, wherein the teacher model is used to generate a target video; S102. Initialize the consistency distillation model to obtain a student model, and transfer the knowledge of the teacher model to the student model through the trajectory segment consistency distillation method; S103. Optimizing the student model by auxiliary loss optimization, dynamic region weighted optimization, and facial detail injection to obtain an optimized model; S104. Input the reference image to be inferred and the human posture sequence into the optimization model to obtain a portrait animation video.

[0035] Specifically, the embodiment of the present application first collects reference image samples and human posture sequence samples through S101 to form a training set. These data are used to train and optimize the model to ensure that the generated video can accurately reflect the target content. Based on the collected training set, a pre-trained teacher model is constructed. The teacher model is responsible for generating the target video. The teacher model has good generation ability and stability, and is used to guide the learning of the student model.

[0036] Secondly, the consistency distillation model is initialized through step S102 to obtain the student model. The student model is a lightweight model that aims to imitate the generation ability of the teacher model. The knowledge of the teacher model is transferred to the student model through the trajectory segmentation consistency distillation method. In this process, the student model learns by imitating the results generated by the teacher model in different time periods to enhance its generation ability.

[0037] Furthermore, through step S103, a series of optimizations are performed on the student model, including the following aspects: Auxiliary loss optimization: Auxiliary loss is introduced to improve the performance of the model during the generation process.

[0038] Dynamic Area Weighted Optimization: Pays special attention to dynamic areas, enhancing the details of these areas to ensure smooth and dynamic animations.

[0039] Facial detail injection: Extract and enhance facial details in generated videos to improve the realism of human expressions and features.

[0040] After the above optimization processing, an optimized student model is finally obtained, which can better generate high-quality portrait animation.

[0041] Specifically, lightweight auxiliary supervision: During the distillation process, the feature distribution of the video latent variable is obtained through the lightweight auxiliary head module and aligned with the real video latent variable; the auxiliary supervision signal is used to enhance the temporal consistency and spatial detail quality of the generated video; Dynamic area focused optimization: Detects dynamic areas in the input video based on inter-frame differences and performs weighted optimization on these areas to reduce blur and distortion in moving areas; Facial fidelity feature injection: extract facial features from the reference image, encode them through a variational autoencoder, and fuse them with global features; use the cross-attention mechanism to model facial features and inject them into the consistency distillation model to improve the realism of facial details in the generated video Finally, in step S104, the reference image to be inferred and the human posture sequence are input into the optimized student model. After being processed by the optimized model, the portrait animation video is output to ensure that it achieves good results in both quality and dynamic performance.

[0042] Optionally, transferring the knowledge of the teacher model to the student model by using the trajectory segment consistency distillation method includes: Divide the probability trajectory into K segments and set the boundary time step of each segment; The objective function is used to optimize self-consistency within each trajectory, and the output results of the teacher model and the student model are compared and optimized so that the student model can continuously imitate the target video generated by the teacher model.

[0043] Optionally, the objective function is:

[0044] in, and are the boundary time steps of two different segments, represents the boundary time step, , is the student model parameter The exponential moving average of For the student model at the boundary time step Input, Based on trajectory points for the teacher model The predicted output.

[0045] Specifically, in this application, in order to optimize the temporal consistency of the generation process, the probability flow trajectory is divided into segments, and set the boundary time step of each segment to , reducing the learning difficulty by optimizing self-consistency within each trajectory. Optionally, the specific process of the auxiliary loss optimization includes: Obtain the true latent variables of the target video generated by the teacher model and the latent variables to be optimized of the predicted video output by the student model; The latent variable to be optimized is compared with the real latent variable to obtain the latent variable difference, and the latent variable difference is optimized by optimizing the auxiliary loss function to improve the temporal consistency and spatial details of the predicted video.

[0046] During the distillation process, a lightweight auxiliary head module is introduced to generate the video latent variables. Make predictions and compare them with the real video latent variables Alignment, improves the temporal consistency and spatial details of the generated video by optimizing the following auxiliary loss function:

[0047] in, is the video latent variable, is the latent variable to be optimized.

[0048] Optionally, the specific process of the dynamic area weighted optimization includes: Determine a target pixel value of a current video frame at a target position, a first pixel value of a first video frame at the target position, and a second pixel value of a second video frame at the target position; the first video frame is a previous frame of the current video frame, and the second video frame is a subsequent frame of the current video frame; Constructing a dynamic region set between video frames of a predicted video output by a student model according to the target pixel value, the first pixel value, the second pixel value and a preset threshold; The dynamic region set and the weighted balance factor are used to obtain the dynamic region loss, and the overall loss between the predicted video and the target video is obtained. The dynamic region loss and the overall loss are dynamically weighted optimized for the predicted video through a dynamic region optimization loss function.

[0049] Specifically, the embodiment of the present application detects the dynamic area by calculating the pixel difference between the input video frames, and the dynamic area set is defined as:

[0050] in, is the preset threshold for dynamic detection, is the pixel coordinate, is the target pixel value of the current video frame, is the second pixel value, is the first pixel value.

[0051] In the dynamic region, the loss is given a higher weight, and the optimization objective function is:

[0052] in, is the overall loss between the predicted video and the target video, is the weighted balancing factor, is the dynamic area loss; Optionally, the specific process of injecting facial details includes: Extracting a facial region of the reference image sample; Encoding the facial region using a variational autoencoder to obtain a high-fidelity facial feature representation; Concatenating and fusing the facial feature representation with the global features of the CLIP image encoder to obtain a fused facial feature; The fused facial features are input into the student model for optimization using a cross-attention mechanism to enhance the fidelity of facial details.

[0053] Specifically, the embodiment of the present application extracts the facial area of ​​the reference image, encodes it through a variational autoencoder, generates a high-fidelity facial feature representation, splices it with the global features of the CLIP image encoder, and uses a cross-attention mechanism to inject the facial features into the consistency distillation model to enhance the realism and consistency of facial details.

[0054] Optionally, the student model is optimized based on a total loss function, and the total loss function is determined based on optimizing an auxiliary loss function and a dynamic region optimization loss function, as shown in the following formula:

[0055]

[0056]

[0057] in, is the overall loss between the predicted video and the target video, is the weighted balancing factor, is the dynamic area loss; is the video latent variable, is the latent variable to be optimized; Weight factor optimized for the auxiliary loss.

[0058] Reference Figure 2 The portrait animation generation method based on consistency distillation provided in the embodiment of the present application specifically includes the following process: Determine the consistency distillation model: pre-train the video diffusion model as the teacher model, the teacher model generates the target video based on the reference image and the human posture sequence; initialize the consistency distillation model as the student model, the student model aims to reduce the reasoning steps, and optimizes it through the consistency distillation method; (2) Trajectory segment consistency distillation: Divide the probability stream trajectory into multiple time periods and optimize the different feature stages of each segment separately; by optimizing the consistency of adjacent trajectory points in the segmented trajectory, the optimization difficulty is reduced and the error accumulation is reduced; (3) Lightweight auxiliary supervision: During the distillation process, the feature distribution of the video latent variable is obtained through the lightweight auxiliary head module and aligned with the real video latent variable. The auxiliary supervision signal is used to enhance the temporal consistency and spatial detail quality of the generated video. (4) Dynamic region focused optimization: Detect dynamic regions in the input video based on inter-frame differences and perform weighted optimization on these regions to reduce blur and distortion in moving areas; (5) Facial fidelity feature injection: extract facial features from the reference image, encode them through a variational autoencoder, and then fuse them with the global features; use the cross-attention mechanism to model facial features and inject them into the consistency distillation model to improve the realism of facial details in the generated video; (6) Reasoning and Generation: Input reference images and human pose sequences, use the optimized student model, and generate high-quality and time-consistent portrait animation videos through 2-4 steps of reasoning.

[0059] Reference Figure 3 This embodiment provides a specific method for generating portrait animation based on consistency distillation, including the following steps: (1) Data collection and model initialization; In order to complete the task of portrait animation generation, it is necessary to first collect appropriate training data, mainly including reference images and human pose sequences. These data will serve as input for the teacher model and the student model. The reference image is static, but contains enough facial and background information, which will provide visual details for the final generated video. The human pose sequence is obtained through key point detection technology, which represents the position information of various parts of the human body. This data is used to guide the character movements in animation generation. In this step, the pre-trained video diffusion model is used as the teacher model. The teacher model already has the ability to generate high-quality videos, so it can generate target videos based on reference images and human pose sequences. This provides a standard generation target for subsequent student model learning.

[0060] (2) Construct a portrait animation generation network with consistent distillation; In this stage, a portrait animation generation network based on consistency distillation is constructed. The network architecture includes the following key modules: Teacher model: A pre-trained video diffusion model used to generate target videos. The teacher model does not need to be optimized during the generation process, and it provides known high-quality video generation capabilities.

[0061] Student model: Compared with the teacher model, the student model has a lighter structure and aims to quickly learn and efficiently generate videos through consistent distillation. The student model is optimized to mimic the process of generating videos by the teacher model.

[0062] Lightweight auxiliary head module: This module is used to help the student model obtain the feature distribution of the video latent variables during the generation process and align them with the latent variables of the real video. By comparing the differences between the generated latent variables and the real latent variables, the performance of the generated video in terms of temporal consistency and spatial details is optimized.

[0063] Dynamic Area Optimization Module: The goal of this module is to focus on optimizing the moving areas in the video, reduce motion blur by weighted processing of the dynamic areas, and ensure the clarity and consistency of the details in the moving areas.

[0064] Facial detail fidelity module: This module extracts facial features from the reference image and encodes them through a variational autoencoder (VAE). The facial features are injected into the student model through a cross-attention mechanism to enhance the fidelity of facial details in the generated video and ensure the realism of facial details.

[0065] (3) Knowledge transfer and generation optimization; In this step, the knowledge of the teacher model is transferred to the student model through the consistency distillation method. The specific operation steps are as follows: (3-1) Trajectory segment consistency distillation: The input video trajectory is divided into multiple time periods, and the model will perform self-consistency optimization in each time period. By optimizing the consistency of trajectory points in each segment, the learning difficulty is reduced and the error accumulation is reduced. This segment distillation method effectively avoids the optimization difficulties in a long time span.

[0066] (3-2) Auxiliary loss optimization: By introducing the auxiliary head module, the student model can predict the latent variables of the video and compare them with the latent variables of the real video. By optimizing the auxiliary loss function, the temporal consistency and spatial details of the video are further improved. This process ensures that the student model can not only imitate the video content generated by the teacher model, but also generate more stable and natural videos.

[0067] (3-3) Weighted optimization of dynamic regions: In the process of generating videos, the model identifies dynamic regions by calculating the pixel differences between video frames. These dynamic regions usually contain parts that move or change rapidly. By weighting and optimizing the dynamic regions, motion blur and distortion are reduced, thus ensuring the clarity and realism of the generated video in the dynamic regions.

[0068] (3-4) Facial detail injection: The facial features in the reference image are encoded by a variational autoencoder and fused with the global features. Then, these facial features are injected into the student model using a cross-attention mechanism to enhance the fidelity of facial details and ensure that the generated video can present high-quality facial details.

[0069] (4) Rapid inference to generate portrait animation; In the inference phase, the model inputs reference images and human pose sequences, and performs fast inference through an optimized student model. Compared with traditional generation methods, the inference process is significantly accelerated after the student model is optimized through consistency distillation, and high-quality and time-consistent dynamic videos can be generated in 2-4 steps. The generated video is not only more realistic in spatial details, but also consistent in time series, providing a smooth animation effect.

[0070] Reference Figure 4 , the present application also provides a portrait animation generation system based on consistency distillation, comprising: A teacher model determination unit 410, used to collect reference image samples and human posture sequence samples as a training set, and to construct a pre-trained teacher model based on the training set, wherein the teacher model is used to generate a target video; A trajectory segmentation consistency distillation unit 420, used to initialize the consistency distillation model to obtain a student model, and transfer the knowledge of the teacher model to the student model through the trajectory segmentation consistency distillation method; An optimization unit 430, configured to optimize the student model by auxiliary loss optimization, dynamic region weighted optimization, and facial detail injection to obtain an optimized model; The inference generation unit 440 is used to input the reference image to be inferred and the human posture sequence into the optimization model to obtain a portrait animation video.

[0071] Optionally, transferring the knowledge of the teacher model to the student model by using the trajectory segment consistency distillation method includes: Divide the probability trajectory into K segments and set the boundary time step of each segment; The objective function is used to optimize self-consistency within each trajectory, and the output results of the teacher model and the student model are compared and optimized so that the student model can continuously imitate the target video generated by the teacher model.

[0072] Optionally, the objective function is:

[0073] in, and are the boundary time steps of two different segments, represents the boundary time step, , is the student model parameter The exponential moving average of For the student model at the boundary time step Input, Based on trajectory points for the teacher model The predicted output.

[0074] Optionally, the specific process of the auxiliary loss optimization includes: Obtain the true latent variables of the target video generated by the teacher model and the latent variables to be optimized of the predicted video output by the student model; The latent variable to be optimized is compared with the real latent variable to obtain the latent variable difference, and the latent variable difference is optimized by optimizing the auxiliary loss function to improve the temporal consistency and spatial details of the predicted video.

[0075] Optionally, the specific process of the dynamic area weighted optimization includes: Determine a target pixel value of a current video frame at a target position, a first pixel value of a first video frame at the target position, and a second pixel value of a second video frame at the target position; the first video frame is a previous frame of the current video frame, and the second video frame is a subsequent frame of the current video frame; Constructing a dynamic region set between video frames of a predicted video output by a student model according to the target pixel value, the first pixel value, the second pixel value and a preset threshold; The dynamic region set and the weighted balance factor are used to obtain the dynamic region loss, and the overall loss between the predicted video and the target video is obtained. The dynamic region loss and the overall loss are dynamically weighted optimized for the predicted video through a dynamic region optimization loss function.

[0076] Optionally, the dynamic region set is defined as:

[0077] in, is the preset threshold for dynamic detection, is the pixel coordinate, is the target pixel value of the current video frame, is the second pixel value, is the first pixel value.

[0078] Optionally, the specific process of injecting facial details includes: Extracting a facial region of the reference image sample; Encoding the facial region using a variational autoencoder to obtain a high-fidelity facial feature representation; Concatenating and fusing the facial feature representation with the global features of the CLIP image encoder to obtain a fused facial feature; The fused facial features are input into the student model for optimization using a cross-attention mechanism to enhance the fidelity of facial details.

[0079] Optionally, the student model is optimized based on a total loss function, and the total loss function is determined based on optimizing an auxiliary loss function and a dynamic region optimization loss function, as shown in the following formula:

[0080]

[0081]

[0082] in, Optimize the loss function for dynamic regions, To optimize the auxiliary loss function; is the overall loss between the predicted video and the target video, is the weighted balancing factor, is the dynamic area loss; is the video latent variable, is the latent variable to be optimized; Weight factor optimized for the auxiliary loss.

[0083] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method, which will not be repeated here.

[0084] Reference Figure 5 Based on the method in the above embodiment, the embodiment of the present application provides an electronic device, which may include: a processor (Processor) 510, a communication interface (Communications Interface) 520, a memory (Memory) 530 and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call the logic instructions in the memory 530 to execute the method in the above embodiment.

[0085] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0086] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method in the above embodiment.

[0087] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method in the above embodiment.

[0088] It is understandable that the processor in the embodiment of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0089] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0090] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0091] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0092] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A portrait animation generation method based on consistency distillation, characterized in that: include: Collect reference image samples and human posture sequence samples as training sets, build a pre-trained teacher model based on the training sets, and use the teacher model to generate a target video; Initialize the consistency distillation model to obtain the student model, and transfer the knowledge of the teacher model to the student model through the trajectory segmentation consistency distillation method; The student model is optimized by auxiliary loss optimization, dynamic region weighted optimization and facial detail injection to obtain an optimized model; The reference image to be inferred and the human posture sequence are input into the optimization model to obtain a portrait animation video.

2. The method for generating portrait animation based on consistency distillation according to claim 1, characterized in that: The method of transferring the knowledge of the teacher model to the student model by using the trajectory segment consistency distillation method includes: Divide the probability trajectory into K segments and set the boundary time step of each segment; The objective function is used to optimize self-consistency within each trajectory, and the output results of the teacher model and the student model are compared and optimized so that the student model can continuously imitate the target video generated by the teacher model.

3. The method for generating portrait animation based on consistency distillation according to claim 2, characterized in that: The objective function is: in, and are the boundary time steps of two different segments, represents the boundary time step, , is the student model parameter The exponential moving average of For the student model at the boundary time step Input, Based on trajectory points for the teacher model The predicted output.

4. The method for generating portrait animation based on consistency distillation according to claim 1, characterized in that: The specific process of auxiliary loss optimization includes: Obtain the true latent variables of the target video generated by the teacher model and the latent variables to be optimized of the predicted video output by the student model; The latent variable to be optimized is compared with the real latent variable to obtain the latent variable difference, and the latent variable difference is optimized by optimizing the auxiliary loss function to improve the temporal consistency and spatial details of the predicted video.

5. The method for generating portrait animation based on consistency distillation according to claim 4, characterized in that: The specific process of the dynamic area weighted optimization includes: Determine a target pixel value of a current video frame at a target position, a first pixel value of a first video frame at the target position, and a second pixel value of a second video frame at the target position; the first video frame is a previous frame of the current video frame, and the second video frame is a subsequent frame of the current video frame; Constructing a dynamic region set between video frames of a predicted video output by a student model according to the target pixel value, the first pixel value, the second pixel value and a preset threshold; The dynamic region set and the weighted balance factor are used to obtain the dynamic region loss, and the overall loss between the predicted video and the target video is obtained. The dynamic region loss and the overall loss are dynamically weighted optimized for the predicted video through a dynamic region optimization loss function.

6. The method for generating portrait animation based on consistency distillation according to claim 5, characterized in that: The dynamic region set is defined as: in, is the preset threshold for dynamic detection, is the pixel coordinate, is the target pixel value of the current video frame, is the second pixel value, is the first pixel value.

7. The method for generating portrait animation based on consistency distillation according to claim 1, characterized in that: The specific process of facial detail injection includes: Extracting a facial region of the reference image sample; Encoding the facial region using a variational autoencoder to obtain a high-fidelity facial feature representation; Concatenating and fusing the facial feature representation with the global features of the CLIP image encoder to obtain a fused facial feature; The fused facial features are input into the student model for optimization using a cross-attention mechanism to enhance the fidelity of facial details.

8. The method for generating portrait animation based on consistency distillation according to claim 5, characterized in that: The student model is optimized based on the total loss function, which is determined based on the optimization auxiliary loss function and the dynamic area optimization loss function, as shown in the following formula: in, Optimize the loss function for dynamic regions, To optimize the auxiliary loss function; is the overall loss between the predicted video and the target video, is the weighted balancing factor, is the dynamic area loss; is the video latent variable, is the latent variable to be optimized; Weight factor optimized for the auxiliary loss.

9. A portrait animation generation system based on consistency distillation, characterized in that: include: A teacher model determination unit, used to collect reference image samples and human posture sequence samples as a training set, and to construct a pre-trained teacher model based on the training set, wherein the teacher model is used to generate a target video; A trajectory segmentation consistency distillation unit, used to initialize the consistency distillation model to obtain a student model, and transfer the knowledge of the teacher model to the student model through the trajectory segmentation consistency distillation method; An optimization unit, used for optimizing the student model through auxiliary loss optimization, dynamic area weighted optimization and facial detail injection to obtain an optimized model; The inference generation unit is used to input the reference image to be inferred and the human posture sequence into the optimization model to obtain a portrait animation video.

10. An electronic device, characterized in that: include: at least one memory for storing a computer program; At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method according to any one of claims 1 to 8.

Images