Audio drive-based portrait animation generation method and device, equipment and medium

Abstract

The present application relates to the technical field of artificial intelligence, and discloses an audio-driven portrait animation generation method, device, equipment and medium, wherein the method comprises: obtaining a reference image, an audio sequence and a text prompt sequence, and performing feature extraction respectively to obtain reference image features, audio features and text feature vectors, and generating text control parameters based on the text feature vectors; based on the reference image features, the audio features and the text control parameters, a low-resolution latent representation sequence is generated frame by frame using an incremental generation method; the low-resolution latent representation sequence is input into a high-resolution enhancement network for time alignment processing to generate a high-resolution latent representation sequence, which is decoded to generate a high-resolution portrait animation video. The present application can be applied to the medical field and the financial technology field, and can maintain the consistency of the identity of the character in long-time continuous generation, and maintain the coherence and stability of the inter-frame details in high-resolution output.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology and can be applied to the medical and financial technology fields. In particular, it relates to an audio-driven portrait animation generation method, device, equipment, and medium. Background Technology

[0002] With the rapid development of artificial intelligence technology, virtual digital human generation technology driven by single portrait images has matured. Its applications have expanded from initial scenarios such as entertainment and social networking, and news broadcasting, to numerous business areas including remote customer service, online education, government services, psychological counseling, and e-commerce live streaming. In these scenarios, virtual digital humans typically need to generate natural-looking, lip-synced, and identity-consistent speaker animations in real time based on preset text or synthesized audio. This replaces or assists real people in providing services such as information broadcasting and interactive guidance, effectively reducing labor costs, improving service standardization, and achieving 24 / 7 uninterrupted response. For example, in remote scenarios in the fintech field, virtual digital humans need to generate natural-looking, lip-synced, and identity-consistent speaker animations in real time based on preset speech or synthesized audio to replace or assist real-person services, thereby reducing operating costs and improving service accessibility. Currently, portrait animation generation technology still faces two prominent bottlenecks when applied to financial and medical scenarios. First, in terms of long-term services, existing methods face serious identity drift and appearance degradation problems. Second, in terms of high-resolution presentation, existing technologies lack effective mechanisms to ensure temporal continuity. High-end financial service terminals and medical telemedicine systems are generally equipped with 4K display devices, requiring the virtual digital human image to be clear, sharp, and with stable details.

[0003] Existing methods, when scaling to higher resolutions, often employ frame-by-frame independent super-resolution processing, ignoring the dimensional dependencies between frames. This easily leads to high-frequency artifacts such as lip texture flickering and eye detail jitter. Such unstable details can exacerbate visual fatigue during prolonged viewing and, in medical scenarios, may interfere with patients' accurate reception and judgment of health information. For example, in remote consultations, eye detail jitter may make it difficult for doctors to observe changes in patient facial expressions, affecting diagnostic accuracy. In summary, current technologies struggle to maintain strict consistency in character identity during long-term continuous generation and cannot maintain the coherence and stability of inter-frame details at 4K high-resolution output. Summary of the Invention

[0004] The purpose of this application is to propose an audio-driven portrait animation generation method, apparatus, device and medium that can strictly maintain the consistency of the person's identity during long-term continuous generation and maintain the coherence and stability of inter-frame details under high-resolution output.

[0005] To address the aforementioned technical problems, embodiments of this application provide an audio-driven portrait animation generation method, including: Acquire a reference image, an audio sequence, and a text prompt sequence, wherein the text prompt sequence is used as text data to guide semantic information in the generation of the portrait animation; Feature extraction is performed on the reference image, the audio sequence, and the text prompt sequence respectively to obtain reference image features, audio features, and text feature vectors, and text control parameters are generated based on the text feature vectors; Based on the reference image features, the audio features, and the text control parameters, a low-resolution latent representation sequence is generated frame by frame using an incremental generation method; The low-resolution latent representation sequence is input into a high-resolution enhancement network for time-alignment processing to generate a high-resolution latent representation sequence. The high-resolution latent representation sequence is decoded to generate a high-resolution portrait animation video.

[0006] To address the aforementioned technical problems, embodiments of this application provide an audio-driven portrait animation generation apparatus, comprising: The initial information acquisition module is used to acquire reference images, audio sequences, and text prompt sequences, wherein the text prompt sequences are used to guide the semantic information text data in the portrait animation generation; The feature extraction module is used to extract features from the reference image, the audio sequence, and the text prompt sequence respectively, to obtain reference image features, audio features, and text feature vectors, and to generate text control parameters based on the text feature vectors; The latent representation sequence generation module is used to generate a low-resolution latent representation sequence frame by frame using an incremental generation method based on the reference image features, the audio features, and the text control parameters. The temporal alignment processing module is used to input the low-resolution latent representation sequence into the high-resolution enhancement network for temporal alignment processing to generate a high-resolution latent representation sequence. A portrait animation video generation module is used to decode the high-resolution latent representation sequence to generate a high-resolution portrait animation video.

[0007] To solve the above-mentioned technical problems, one technical solution adopted by the present invention is to provide a computer device, including one or more processors; and a memory for storing one or more programs, such that the one or more processors implement the audio-driven portrait animation generation method described above.

[0008] To solve the above-mentioned technical problems, one technical solution adopted by the present invention is: a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the audio-driven portrait animation generation method described above.

[0009] This invention provides an audio-driven portrait animation generation method, apparatus, device, and medium. It acquires a reference image, an audio sequence, and a text prompt sequence, wherein the text prompt sequence is used to guide semantic information in the portrait animation generation. Feature extraction is performed on the reference image, audio sequence, and text prompt sequence to obtain reference image features, audio features, and text feature vectors, and text control parameters are generated based on the text feature vectors. Based on the reference image features, audio features, and text control parameters, a low-resolution latent representation sequence is generated frame-by-frame using an incremental generation method. The low-resolution latent representation sequence is input into a high-resolution enhancement network for temporal alignment processing to generate a high-resolution latent representation sequence. The high-resolution latent representation sequence is decoded to generate a high-resolution portrait animation video. This effectively suppresses identity drift during long-term generation while ensuring detail stability under high-resolution output. It maintains the consistency of character identity during long-term continuous generation and preserves the coherence and stability of inter-frame details under high-resolution output. Attached Figure Description

[0010] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram of an application environment for an audio-driven portrait animation generation method according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating the implementation of the audio-driven portrait animation generation method provided in this application embodiment; Figure 3 yes Figure 2 A flowchart illustrating a specific implementation method of step S2; Figure 4 yes Figure 2 A flowchart illustrating a specific implementation method of step S3; Figure 5 yes Figure 4 A flowchart illustrating a specific implementation of step S32; Figure 6 yes Figure 4A flowchart illustrating a specific implementation of step S33; Figure 7 yes Figure 2 A flowchart illustrating a specific implementation of step S4; Figure 8 This is a schematic diagram of an audio-driven portrait animation generation device provided in an embodiment of this application; Figure 9 This is a schematic diagram of the computer device provided in the embodiments of this application. Detailed Implementation

[0012] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0013] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0014] Traditional portrait animation generation techniques, in long-term services, often suffer from cumulative errors in previous frames due to incremental generation mechanisms. This leads to facial features deviating from the original reference image, resulting in identity drift and appearance degradation. Furthermore, at high resolutions, existing methods often employ independent super-resolution frame by frame, ignoring inter-frame dependencies. This easily produces high-frequency artifacts such as flickering lip textures and jittering eye details, making it difficult to maintain the continuity and stability of details between frames. This severely restricts the deep application of virtual digital humans in the financial and medical fields.

[0015] To address this, this application proposes an audio-driven portrait animation generation method, comprising: acquiring a reference image, an audio sequence, and a text prompt sequence, wherein the text prompt sequence is used to guide the semantic information in the portrait animation generation; extracting features from the reference image, audio sequence, and text prompt sequence respectively to obtain reference image features, audio features, and text feature vectors, and generating text control parameters based on the text feature vectors; generating a low-resolution latent representation sequence frame by frame using an incremental generation method based on the reference image features, audio features, and text control parameters; inputting the low-resolution latent representation sequence into a high-resolution enhancement network for temporal alignment processing to generate a high-resolution latent representation sequence; and decoding the high-resolution latent representation sequence to generate a high-resolution portrait animation video.

[0016] For ease of understanding, the following explains some key terms in this embodiment: Reference image: A static image used to provide a baseline for the identity and appearance of a person in portrait animation generation. This image carries visual information such as the target person's facial structure, skin color, and hairstyle, serving as an identity anchor point in the subsequent animation generation process.

[0017] Audio sequence: refers to the continuous sound data that drives lip movements and facial expressions in portrait animation. This sequence typically contains speech information, and its rhythm, intonation, and content will affect the dynamic performance of the generated animation.

[0018] Text cue sequences: These are text data used to guide semantic information such as expressions and postures in portrait animation generation. These sequences provide high-level semantic instructions, enabling the generated animation to express specific emotions or action intentions.

[0019] Reference image features: These are numerical representations extracted from reference images to characterize a person's identity and appearance. These features capture key visual information from the image to ensure consistency between the generated animation and the reference character.

[0020] Audio features: These are numerical representations extracted from audio sequences that drive facial movements and expressions. These features typically include information such as pitch, volume, and speech rate, reflecting the dynamic changes in speech.

[0021] Text feature vector: A numerical representation extracted from a sequence of text prompts to encode semantic information. This vector transforms the text content into a machine-processable form to control the expressions and postures of animations.

[0022] Text control parameters: These are dynamic parameters generated based on text feature vectors and used for fine-tuning the facial expressions and / or postures of animated portraits. These parameters provide semantic-level control over the animation's performance.

[0023] Incremental generation: refers to a method of generating video sequences frame by frame, in which the generation of the current frame uses information from the previous frame or preceding frame as a condition to maintain the continuity between frames.

[0024] Low-resolution latent representation sequence: refers to a sequence of video frames represented in a low-dimensional latent space that has not yet been decoded into pixel images. This sequence contains core information such as motion and facial expressions in animation, but with low resolution.

[0025] High-resolution enhancement networks: These are neural network models used to upscale low-resolution latent representation sequences to high resolution and perform temporal optimization. The aim of this network is to enhance image details and ensure temporal coherence between frames.

[0026] Temporal alignment processing: In high-resolution enhancement networks, video sequences are processed along the time dimension to ensure smooth transitions and consistency in visual content and motion trajectories between adjacent frames.

[0027] High-resolution latent representation sequence: refers to a sequence of video frames that, after being processed by a high-resolution enhancement network, possess higher detail and temporal coherence in the latent space. This sequence prepares the video for final high-resolution decoding.

[0028] Decoding: refers to the process of converting representations in latent space (such as latent representation sequences) into pixel-space images or videos that can be viewed directly.

[0029] High-resolution portrait animation video: refers to the final output portrait animation video with clear details and smooth motion, whose resolution meets the requirements for high-quality display.

[0030] This application effectively solves the problems of identity drift and appearance degradation in long-duration portrait animation generation by using a multimodal information collaborative driving and phased generation strategy. This ensures that the virtual digital human maintains strict consistency in identity even when broadcasting content for extended periods in scenarios such as financial customer service or medical guidance. Simultaneously, through temporal alignment processing in the high-resolution enhancement network, it overcomes the flickering and jitter between frames in high-resolution output, achieving clear, sharp, and stable high-resolution portrait animation videos, thus enhancing user trust and the authority of the information conveyed.

[0031] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0032] It should be noted that the audio-driven portrait animation generation method provided in this application embodiment is generally executed by a server, and correspondingly, the audio-driven portrait animation generation device is generally configured in the server.

[0033] The audio-driven portrait animation generation method provided in this invention can be applied to, for example... Figure 1In this application environment, the client communicates with the server via a network. The server can receive reference images, audio sequences, and text prompt sequences from the client; and generate a high-resolution portrait animation video based on the reference images, audio sequences, and text prompt sequences. In this invention, the server sends a pre-authorization decision to the client. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0034] The audio-driven portrait animation generation method provided in this application can be applied to online consultation and digital therapy scenarios in the medical field and remote banking and intelligent investment advisory scenarios in the fintech field.

[0035] Please see Figure 2 , Figure 2 A specific implementation of an audio-driven portrait animation generation method is shown.

[0036] It should be noted that if substantially the same result is obtained, the method of this invention is not based on... Figure 2 Limited to the order of the processes shown, this method includes the following steps: S1: Obtain a reference image, an audio sequence, and a text prompt sequence, wherein the text prompt sequence is used to guide the semantic information text data in the portrait animation generation.

[0037] Specifically, the reference image can be obtained by the user through a photograph or scan using an image acquisition device, or it can be selected from a preset image library. The audio sequence can be recorded in real time by a microphone, or read from a preset audio file. The text prompt sequence can be entered by the user via a keyboard, or speech can be converted into text using speech recognition technology. For example, in a financial customer service scenario, the reference image could be a portrait of a virtual customer service representative, the audio sequence could be a preset welcome message, and the text prompt sequence could be instructions such as "smile" or "nod".

[0038] S2: Extract features from the reference image, the audio sequence, and the text prompt sequence respectively to obtain reference image features, audio features, and text feature vectors, and generate text control parameters based on the text feature vectors.

[0039] Specifically, for feature extraction of the reference image, traditional image processing algorithms, such as edge detection and texture analysis, can be used to obtain low-level visual features of the image. For feature extraction of the audio sequence, methods such as Fourier transform and Mel-frequency cepstral coefficients (MFCC) can be used to obtain the spectral or temporal features of the audio. For text prompt sequences, methods such as the Bag-of-Words model or TF-IDF can be used to convert them into text feature vectors. When generating text control parameters based on text feature vectors, simple linear mapping or lookup table methods can be used to map the text feature vectors to parameters used to control facial expressions or postures. For example, when the text feature vector represents "happy," it can be mapped to a set of preset facial expression parameters. The text control parameters include scaling parameters and offset parameters.

[0040] Please see Figure 3 , Figure 3 A specific implementation of step S2 is shown below: S21: Extract multi-scale visual features from the reference image using an image encoder to obtain the reference image features. S22: Extract the audio embedding of each frame from the audio sequence using an audio encoder to generate the audio features. S23: Convert the text prompt sequence into the text feature vector using a text encoder, and generate the text control parameters frame-by-frame based on the text feature vector using a multilayer perceptron.

[0041] Image encoders aim to extract visual information from input reference images that represents a person's identity, facial structure, and texture details. Their "multi-scale" nature means the encoder can capture features at different granularities; for example, shallow features might focus on edges and textures, while deeper features encode higher-level semantic information, such as the shape and relative position of facial organs. Specifically, image encoders can employ pre-trained convolutional neural network (CNN) architectures, such as ResNet, VGG, or EfficientNet. These networks, through multiple layers of convolution and pooling operations, can effectively extract feature maps with rich semantic and spatial information from images. Another approach is to use Transformer-based visual models, such as Vision Transformer (ViT) or Swin Transformer. These models, through self-attention mechanisms, can capture long-range dependencies in images, thus obtaining more comprehensive multi-scale visual features.

[0042] Audio encoders are used to transform input audio sequences into a series of discrete audio embeddings with semantic information. These embeddings capture the phonemes, intonation, emotion, and unique acoustic features of the speaker, providing precise driving signals for subsequent lip-syncing and facial expression-driven processing. Specifically, audio encoders can employ recurrent neural network (RNN) structures, such as Long Short-Term Memory (LSTM) networks or gated recurrent units (GRUs), which excel at processing sequential data and capturing temporal dependencies in audio sequences. Alternatively, Transformer-based audio models, such as Wav2Vec 2.0 or HuBERT, can be used. These models are pre-trained on large-scale speech data through self-supervised learning, enabling the generation of high-quality, context-aware audio embeddings.

[0043] The text encoder is responsible for converting the input sequence of text prompts (e.g., text instructions describing expressions or gestures) into a high-dimensional text feature vector containing semantic information. The Multilayer Perceptron (MLP) then maps this static text feature vector to dynamic, frame-by-frame text control parameters, which can be directly used to adjust the expressions and gestures of the portrait animation. Specifically, the text encoder can employ a pre-trained Language Model (PLM), such as BERT, GPT, or T5, which can understand complex language structures and semantics to generate expressive text feature vectors. The MLP can be a neural network consisting of multiple fully connected layers, whose input is the text feature vector and output is the text control parameters required for each frame of animation. To achieve frame-by-frame generation, the MLP can be designed to receive time-step information as additional input, or it can use a sequence generation module (such as another RNN or Transformer decoder) to expand the text feature vector into serialized control parameters.

[0044] The embodiments of this application provide a structured, high-fidelity, and controllable multimodal input for the subsequent generation of low-resolution latent representation sequences, which greatly improves the generation quality and controllability of portrait animations and provides reliable technical support for application scenarios such as finance and healthcare that have extremely high requirements for accuracy and stability.

[0045] S3: Based on the reference image features, the audio features, and the text control parameters, a low-resolution latent representation sequence is generated frame by frame using an incremental generation method.

[0046] Specifically, in the incremental generation process, the latent representation of the previous frame can be directly used as the motion condition for the current frame, and then concatenated or weighted with reference image features, the corresponding audio features of the current frame, and text control parameters. This concatenation is then input into a generative network to generate the latent representation of the current frame. For example, when generating the Nth frame, the latent representation of the (N-1)th frame can be directly used, combined with the audio and text information of the Nth frame, to generate the latent representation of the Nth frame through a convolutional neural network. This approach maintains basic temporal continuity but may accumulate small errors or distortions present in the previous frame.

[0047] Please see Figure 4 , Figure 4 A specific implementation of step S3 is shown below: S31: For each current frame to be generated, obtain the latent representation of the preceding generated frame, wherein the current frame is the target frame being generated at the current moment in the incremental generation method, and the preceding generated frame is the frame before the current frame. S32: Perform anti-appearance contamination enhancement processing on the latent representation of the preceding generated frame to obtain enhanced motion conditions. S33: Based on the enhanced motion conditions, the reference image features, the currently corresponding audio features, and the currently corresponding text control parameters, generate the low-resolution latent representation sequence.

[0048] Specifically, when generating each current frame, the latent representation of the preceding generated frame must first be obtained. This step aims to provide an initial motion or state reference for the generation of the current frame. The latent representation is an abstract encoding of an image or video in low-dimensional space, containing key visual information such as facial structure, expression, and pose. By obtaining the latent representation of the preceding generated frame, the temporal continuity of the animation can be ensured, allowing the generation of the current frame to inherit the motion trends and appearance information of the previous frame, avoiding jumps or discontinuities in the animation.

[0049] Subsequently, the latent representation of the preceding generated frame undergoes anti-appearance contamination enhancement processing. This processing is one of the core innovations of this application, aiming to eliminate or mitigate accumulated errors, noise, or unexpected appearance features that may exist in the latent representation of the preceding frame, thereby preventing these "contaminations" from propagating and amplifying frame by frame during incremental generation. This enhancement processing ensures that the motion conditions input into the current frame generation process are pure and stable, thus effectively suppressing identity drift and appearance degradation. Specific implementation methods may include, but are not limited to: reconstructing the latent representation of the preceding generated frame using a pre-trained autoencoder or denoising network, and introducing a correction mechanism during the reconstruction process, for example, using reference image features to constrain the reconstruction result to ensure that the reconstructed latent representation is closer to the original identity features while removing noise; or, applying specific random perturbations to the latent representation of the preceding generated frame and combining it with regularization techniques to encourage the model to not overly rely on the local details of the previous frame when generating the current frame, but to focus more on global, stable identity features and driving signals, for example, the latent representation of the preceding generated frame can be divided into multiple image blocks, and some of the image blocks can be randomly set to zero according to a preset probability, and then Gaussian noise can be added to the latent representation after the block discarding process; or, combining the latent representations of multiple preceding frames, using temporal fusion or filtering techniques to smooth the instantaneous fluctuations and noise in the latent representation, thereby obtaining a more stable and representative motion condition. After anti-appearance contamination enhancement processing, the resulting enhanced motion condition is a purified and stabilized latent representation that includes motion information from the previous frame but excludes accumulated errors and noise that could cause identity drift or appearance degradation. This enhanced motion condition will serve as a crucial input driving the generation of the current frame, ensuring the stability and identity consistency of subsequent generation processes.

[0050] Finally, based on the enhanced motion conditions, the reference image features, the currently corresponding audio features, and the currently corresponding text control parameters, the latent representation of the current frame is generated. This step is the core of multimodal information fusion and generation. Enhanced motion conditions provide corrected motion trajectory information, ensuring the continuity of the animation. The reference image features serve as identity anchors, continuously pulling the generated content back to the identity features of the original portrait, preventing facial features from deviating. The currently corresponding audio features (such as pitch, volume, and speech rate) are used to drive subtle changes in lip shape and facial expressions, achieving lip-sound synchronization. The text control parameters provide a higher level of semantic control, used to adjust the intensity of expressions or the amplitude of postures to achieve richer expressiveness. By effectively fusing these multi-source information sources in the generative network, the generation of the latent representation of the current frame can be comprehensively driven, thereby achieving high-quality, high-fidelity portrait animation.

[0051] The synergistic effect of this multimodal condition in the embodiments of this application enables the generation of the current frame to maintain the motion continuity with the previous frame while effectively avoiding identity drift caused by the cumulative error of the previous frame. Thus, high-quality and high-stability portrait animation output is achieved under the incremental generation paradigm, solving the problems of identity drift and appearance degradation faced by the prior art in long-term services.

[0052] Please see Figure 5 , Figure 5 A specific implementation of step S32 is shown below: S321: Divide the latent representation of the preceding generated frame into multiple image blocks, and randomly set some of the image blocks to zero according to a preset probability. S322: Add Gaussian noise to the latent representation after block discarding to obtain the enhanced motion conditions.

[0053] Specifically, the latent representation of the previous frame is spatially divided into several independent regions or units. This aims to provide a foundation for subsequent local information processing, enabling the model to perform targeted operations on different parts of the latent representation, rather than processing it holistically. For example, the latent representation can be divided into regular grid-like image blocks, such as uniform 4x4 or 8x8 blocks; or, a semantic-based segmentation method can be used to identify and extract feature blocks corresponding to key facial regions (such as eyes, nose, and mouth) from the latent representation through a pre-trained segmentation network.

[0054] In the segmented image blocks, a subset of blocks is randomly selected based on pre-defined probability values, and all values ​​within these blocks are set to zero. This operation simulates situations of local information loss or occlusion, aiming to force the generative model to reduce its over-reliance on specific local details from previous frames when generating the current frame. For example, a random number can be generated independently for each image block, and if this random number is less than a preset probability, the image block is set to zero; alternatively, the number of image blocks that need to be zeroed can be calculated based on a preset probability, and then a corresponding number of image blocks can be randomly selected from all image blocks for zeroing.

[0055] After some image patches are zeroed out, random noise conforming to a Gaussian distribution is introduced into the remaining latent representation. This step aims to further perturb the motion conditions, enhance the robustness of the model, and break the deterministic propagation path of errors. For example, Gaussian noise with zero mean and a fixed variance can be generated and superimposed element-wise onto the latent representation after patch discarding; alternatively, the variance of the Gaussian noise can be dynamically adjusted based on the current generation step size or the feature intensity of the latent representation to achieve adaptive noise addition.

[0056] This application embodiment achieves effective blocking of error propagation during incremental generation by actively denoising and reconstructing motion conditions. This ensures that the model can always maintain the fidelity of the identity features of the original reference image in long-term continuous generation tasks, and effectively avoids the gradual distortion of facial features in cyclic generation, thereby solving the problems of identity drift and appearance degradation in long-term services.

[0057] Please see Figure 6 , Figure 6 A specific implementation of step S33 is shown below: S331: Concatenate the enhanced motion conditions with the noise latent variables of the current denoising step along the channel dimension to obtain noise latent variables with fused motion information. S332: Input the noise latent variables with fused motion information into the denoising network. Through the cross-attention layer of the denoising network, use the reference image features and the currently corresponding audio features as conditions, and inject features into the noise latent variables with fused motion information based on the conditions to obtain an intermediate feature map with fused multimodal conditions. S333: Perform adaptive layer normalization modulation on the intermediate feature map using the currently corresponding scaling parameters and offset parameters to obtain a modulated feature map, thereby controlling facial expressions and / or postures. S334: Denoise the modulated feature map to obtain new latent variables output by the current denoising step. S335: Use the new latent variables output by the current denoising step as the noise latent variables for the next denoising step, and repeat the denoising step iteration process until a preset number of denoising steps is reached, generating the low-resolution latent representation sequence.

[0058] Specifically, the enhanced motion conditions are concatenated with the noise latent variables of the current denoising step along the channel dimension to obtain noise latent variables that fuse motion information. The enhanced motion conditions are obtained by enhancing the latent representation of the previous frame with anti-appearance contamination (e.g., by dividing the latent representation into multiple image blocks, randomly zeroing them, and then adding Gaussian noise). They contain motion information from the previous frame and have been processed to reduce accumulated errors. The noise latent variables of the current denoising step are the noisy latent representations received by the denoising network in the current iteration step, representing the input of the current denoising stage. Concatenating along the channel dimension means connecting these two feature tensors along the channel dimension. For example, if the enhanced motion conditions have C1 channels and the noise latent variables have C2 channels, the concatenation will form a tensor with C1 + C2 channels. This operation allows the denoising network to simultaneously receive and process information from the motion conditions and the current noise latent variables in the early stages of the denoising process, laying the foundation for subsequent feature evolution.

[0059] Secondly, the noisy latent variables of the fused motion information are input into a denoising network. Through the cross-attention layer of this network, the reference image features and the corresponding audio features are used as conditions. Based on these conditions, features are injected into the noisy latent variables of the fused motion information to obtain an intermediate feature map with fused multimodal conditions. The denoising network typically employs a U-Net architecture based on a diffusion model (such as DDPM or Latent Diffusion Model) or other generative models. Its core task is to recover a clear latent representation from noise. The cross-attention layer is an attention mechanism in the Transformer architecture, allowing the model to focus on relevant information in another sequence (such as reference image features and audio features) while processing one sequence (e.g., the noisy latent variables of fused motion information). Through this mechanism, the reference image features (providing identity information) and the audio features (providing lip-sync and emotion-driven information) are effectively injected into the main processing path, ensuring that the generated portrait animation maintains identity consistency while possessing dynamic expressiveness synchronized with the audio.

[0060] Next, adaptive layer normalization modulation (AdaLN) is performed on the intermediate feature map using the currently corresponding scaling and offset parameters to obtain a modulated feature map, thereby enabling control over expressions and / or postures. The scaling parameters adjust the amplitude or intensity of the feature map, thus affecting the "intensity" or "amplitude" of the generated portrait animation's expression or posture; for example, it can make the expression more exaggerated or the posture change more pronounced. The offset parameters adjust the baseline or bias of the feature map, thus affecting the "direction" or "starting point" of the generated portrait animation's expression or posture; for example, it can make the facial expression more inclined towards a smile or seriousness. The adaptive layer normalization modulation (AdaLN) is a conditional injection technique that modulates the output of layer normalization using learned scaling and offset factors, thereby finely controlling the style or attributes of the generated content. Here, the scaling and offset factors are obtained by mapping the text control parameters (scaling and offset parameters). By dynamically adjusting the statistical characteristics of the feature map, AdaLN can specifically correct and enhance the generated facial movements based on the semantic content of the text prompts, thereby achieving precise control over expressions and / or postures.

[0061] Subsequently, the modulated feature map is denoised to obtain new latent variables output from the current denoising step. The modulated feature map is then denoised again by subsequent layers of the denoising network, which refer to the parts of the denoising network located after the normalization modulation of the cross-attention layer and the adaptive layer. These layers continue to process the modulated feature map to further remove noise and refine the latent representation. The latent variables output from the current denoising step are the model's prediction of the noisy latent variables after the current denoising step, and they are closer to the final noise-free latent representation than the input noisy latent variables.

[0062] Finally, the latent variable output from the new current denoising step is used as the noise latent variable for the next denoising step, and the denoising step iteration process is repeated until the preset number of denoising steps is reached to generate the latent representation of the current frame. Diffusion models typically require multiple iterations (e.g., tens to hundreds of steps) to completely denoise random noise into a clear image latent representation. This iterative generation method ensures that the latent representation of each frame gradually stabilizes and becomes natural under the constraints of multimodal conditions and text control.

[0063] This application combines multimodal conditional fusion with text-driven control, which significantly improves the naturalness, expressiveness, and identity consistency of portrait animation during iterative denoising. Especially in long-term generation, it can effectively avoid problems such as facial features deviating from the original reference image and facial feature distortion, thus providing a high-quality solution for application scenarios such as finance and healthcare that have extremely high requirements for the stability of virtual digital humans.

[0064] S4: Input the low-resolution latent representation sequence into the high-resolution enhancement network for time alignment processing to generate a high-resolution latent representation sequence.

[0065] Specifically, the high-resolution enhancement network can consist of a series of convolutional layers and upsampling layers to progressively increase the resolution of the latent representation sequence. For temporal alignment, simple inter-frame interpolation algorithms, such as optical flow, can be used to interpolate adjacent frames to generate intermediate frames, thus smoothing the inter-frame transition to some extent. For example, for two adjacent low-resolution latent representation frames, the optical flow between them can be calculated, and one frame can be warped based on the optical flow information to align it with the other frame. Then, they can be fused to generate a higher-resolution latent representation.

[0066] The high-resolution enhancement network consists of multiple spatiotemporal Transformer blocks stacked together, each of which includes a spatial self-attention layer and a temporal alignment layer.

[0067] Please see Figure 7 , Figure 7 A specific implementation of step S4 is shown below: S41: In the spatial self-attention layer, spatial self-attention calculation is performed independently on each frame latent representation in the low-resolution latent representation sequence to obtain an intermediate latent representation sequence with enhanced spatial details. S42: The intermediate latent representation sequence with enhanced spatial details is input into the temporal alignment layer. In the temporal alignment layer, attention calculation and smoothing are performed on features at the same spatial location along the temporal dimension of the intermediate latent representation sequence to obtain the temporally coherent high-resolution latent representation sequence.

[0068] Specifically, the spatial self-attention layer is an attention mechanism that calculates the interdependencies between different spatial locations within a single image frame, thereby enhancing local details and global consistency. Through this mechanism, the network can identify and enhance important regions in the image while suppressing irrelevant or noisy regions. Besides the standard Transformer self-attention mechanism, the spatial self-attention layer can also be implemented through local self-attention (such as window attention), multi-head self-attention mechanisms, or self-attention combined with convolutional operations (such as the ContextualTransformer Block in CoTNet). The core function of this step is to perform fine-grained spatial feature extraction and enhancement on each frame of the input low-resolution latent representation sequence, fully capturing and enhancing the texture, edge, and structural information within a single frame, providing a high-quality, detailed spatial foundation for subsequent temporal processing. Specifically, each frame of latent representation can be treated as an independent image feature map, which is then input into the spatial self-attention layer. This layer calculates the attention weights between each pixel (or feature point) and other pixels within the feature map, and weights and aggregates the features based on these weights, thereby generating an intermediate latent representation sequence that is more detailed and expressive in the spatial dimension.

[0069] The temporal alignment layer aims to handle the temporal relationships between frames in a video sequence, ensuring the video's coherence and smoothness. It compensates for inter-frame motion and deformation by manipulating features along the temporal dimension, thereby eliminating flicker or jitter that may occur from processing each frame independently. The temporal alignment layer can employ various techniques, such as optical flow-based feature alignment, deformable convolution for feature sampling, or using recurrent neural networks (RNNs) or 3D convolutional networks to model temporal dependencies. The main function of this step is to ensure the temporal coherence and stability of the video sequence. Even after spatial detail enhancement, although the quality of individual frames is improved, inconsistencies may still exist between frames. The temporal alignment layer eliminates artifacts such as jitter and flicker by explicitly modeling inter-frame relationships and performing attention calculations and smoothing on features at the same spatial location along the temporal axis, generating a visually smooth and coherent high-resolution latent representation sequence. For example, a temporal alignment layer can receive features from the same spatial location from multiple adjacent frames, then learn the importance of these features through an attention mechanism, and perform weighted averaging or interpolation to generate a more temporally smoother feature. Another approach is to utilize 3D convolutional kernels for feature aggregation and filtering in the temporal dimension, or to use recurrent units (such as GRUs or LSTMs) to pass and update temporal information to achieve feature smoothing and alignment.

[0070] The embodiments of this application effectively solve the high-frequency artifact problems such as lip texture flickering and eye detail jitter caused by the lack of an effective mechanism to ensure temporal coherence when mapping low-resolution latent representations to high-resolution space.

[0071] S5: Decode the high-resolution latent representation sequence to generate a high-resolution portrait animation video.

[0072] Specifically, the decoding process can be performed by a standard image decoder, such as a deconvolutional network, which upsamples each frame of the latent representation sequence and converts it into an image in pixel space. For example, each frame of the high-resolution latent representation sequence can be input into a pre-defined decoder, which, through a series of deconvolutional layers and activation functions, converts the abstract latent representation into image frames with specific pixel values, ultimately combining them into a high-resolution portrait animation video.

[0073] Further, step S5 includes: inputting each frame of latent representation in the high-resolution latent representation sequence into a pre-trained decoder for upsampling and decoding into the high-resolution portrait animation video.

[0074] Specifically, a high-resolution latent representation sequence refers to a sequence of feature representations output by a high-resolution enhancement network that compresses and abstracts the content of a high-resolution image in a low-dimensional latent space. Each frame of the latent representation in this sequence contains the key information needed to generate the corresponding high-resolution image, but it has not yet been converted to pixel format. For example, the high-resolution latent representation sequence can be a series of tensors, each representing a latent feature of a frame of image, typically having channel, height, and width dimensions, but with a resolution lower than the final target pixel space. Alternatively, the high-resolution latent representation sequence can be a series of feature maps generated by an encoder of a variational autoencoder (VAE) or a diffusion model, which capture the semantic and stylistic information of the high-resolution frame.

[0075] A pre-trained decoder is a neural network module trained on a large amount of data, whose function is to inversely transform latent representations into high-resolution images. Through learned mapping relationships, this decoder can transform abstract latent features into pixel-level images with rich visual details. For example, the pre-trained decoder can be the decoder part of a pre-trained variational autoencoder (VAE) or generative adversarial network (GAN), models that excel at generating realistic images from latent codes. Furthermore, the pre-trained decoder can also be the decoder of a pre-trained diffusion model, designed to progressively refine noisy latent representations into sharp images.

[0076] Upsampling is a crucial step within the decoder, designed to increase the spatial resolution of the feature map or image. This process is essential for expanding the compact latent representation to the desired high-resolution pixel space. For example, upsampling can be achieved through transposed convolution (also known as deconvolution), a learnable upsampling method that increases spatial dimensions by performing a convolution-like operation while learning appropriate feature transformations. Another implementation uses nearest-neighbor interpolation, a simple method that fills a new grid by copying pixel values, typically followed by convolutional layers for refinement. Decoding is the overall process performed by the pre-trained decoder to convert the latent representation into a pixel-space image. It encompasses upsampling, feature transformation, and the usual application of activation functions to generate the final pixel values. For example, the decoding process can consist of a series of convolutional layers, batch normalization layers, and activation functions (such as ReLU, Sigmoid, Tanh) that progressively convert the upsampled feature map into a multi-channel image.

[0077] The decoding method in this application not only ensures the continuity of video generation, but also ensures that the final output portrait animation has clear texture and stable structure in visual presentation through independent decoding and upsampling of each frame. This enables the generated virtual digital human to present a clear, sharp, and stable visual effect on high-resolution display devices, thereby meeting the stringent requirements of high-quality virtual digital human interaction in fields such as finance and healthcare, and improving user experience and trust.

[0078] This application provides a specific example to illustrate the above technical solution in more detail: In a fintech application scenario, a bank needs to deploy a virtual digital avatar as an intelligent investment advisor to provide long-term online consultation services to user A. This virtual digital avatar needs to possess highly realistic facial expressions, precise lip-sync, and maintain stable identity characteristics during extended conversations, while simultaneously presenting clear and coherent visuals on a 4K display device.

[0079] First, the system acquires a reference image of the virtual digital human, an audio sequence corresponding to the voice content of the robo-advisor, and a text prompt sequence for controlling specific expressions or postures. For example, the reference image could be a standard portrait of the virtual digital human, the audio sequence could be a pre-recorded or real-time synthesized introduction to financial products, and the text prompt sequence might contain instructions such as "smile when mentioning returns."

[0080] Next, the system extracts features from these inputs. An image encoder extracts multi-scale visual features from reference images; these features encode the unique identity information and appearance details of the virtual digital human. Simultaneously, an audio encoder processes the audio sequence, extracting audio embeddings frame by frame to capture the prosody and pronunciation information of the speech. A text encoder converts the text cue sequence into a text feature vector. Subsequently, a multilayer perceptron generates text control parameters frame by frame based on this text feature vector. These parameters include scaling and offset parameters for subsequent fine-grained control of facial expressions and posture.

[0081] Subsequently, the system generates a low-resolution latent representation sequence frame by frame using an incremental generation method. This process aims to address the identity drift and appearance degradation problems existing in long-term services in current technologies. For each current frame to be generated, the system first obtains the latent representation of the previously generated frame. To avoid potential error accumulation and appearance contamination in the previous frame, the system performs anti-appearance contamination enhancement processing on the latent representation of the previously generated frame. Specifically, the latent representation of the previously generated frame is divided into multiple image blocks, and some image blocks are randomly set to zero according to a preset probability to destroy any possible accumulated distortion. Then, Gaussian noise is added to the latent representation after block discarding to further blur the precise appearance information of the previous frame, thereby obtaining enhanced motion conditions.

[0082] Based on this, the system generates the latent representation of the current frame using enhanced motion conditions, reference image features, currently corresponding audio features, and currently corresponding text control parameters. The specific process is as follows: The enhanced motion conditions and the noise latent variables from the current denoising step are concatenated along the channel dimension to obtain noise latent variables that fuse motion information. This variable is input into the denoising network. Through the cross-attention layer of the denoising network, the reference image features and currently corresponding audio features are injected as conditions to inject features into the noise latent variables that fuse motion information, resulting in an intermediate feature map fused with multimodal conditions. Subsequently, adaptive layer normalization modulation is performed on the intermediate feature map using the currently corresponding scaling and offset parameters to achieve precise control over the virtual digital human's expressions and postures, such as generating specific lip shapes or head movements based on text prompts. Based on the modulated feature map, subsequent layers of the denoising network complete the denoising processing of the current denoising step, obtaining new latent variables output by the current denoising step. After all denoising steps are iterated, the latent representation of the current frame is finally generated. By combining anti-appearance contamination enhancement processing and multimodal conditional injection in this incremental generation method, this approach effectively suppresses identity drift, ensures the stability of facial features of virtual digital humans during long-term consultations, and avoids problems such as facial feature deformation and loss of identity features commonly found in existing technologies.

[0083] After generating the low-resolution latent representation sequence, to address the lack of temporal coherence in existing technologies when rendering at high resolution, this sequence is input into a high-resolution enhancement network for temporal alignment processing. The high-resolution enhancement network consists of multiple stacked spatiotemporal Transformer blocks, each including a spatial self-attention layer and a temporal alignment layer. In the spatial self-attention layer, spatial self-attention computation is performed independently on the latent representation of each frame in the low-resolution latent representation sequence to enhance the spatial details of each frame, such as facial textures and hair details. Subsequently, the intermediate latent representation sequence with enhanced spatial details is input into the temporal alignment layer. In the temporal alignment layer, attention computation and smoothing are performed on features at the same spatial location along the temporal dimension of the intermediate latent representation sequence. Unlike the frame-by-frame independent super-resolution processing in existing technologies, this method explicitly captures and utilizes inter-frame dependencies through the temporal alignment layer, effectively eliminating high-frequency artifacts such as lip texture flicker and eye detail jitter, ensuring the temporal coherence and stability of the high-resolution portrait animation video. Finally, a high-resolution latent representation sequence is generated.

[0084] Finally, the high-resolution latent representation sequence is decoded to generate a high-resolution portrait animation video. Specifically, each frame of the latent representation sequence is input into a pre-trained decoder for upsampling and decoding. The decoder converts the latent representation into a high-resolution image in pixel space, ultimately forming a high-quality, identity-stable, and temporally coherent virtual digital human animation video that user A watches in a financial scenario.

[0085] This application maintains the consistency of the virtual digital human identity during long-term services and preserves the coherence and stability of inter-frame details under high-resolution output, overcoming the main bottlenecks of existing technologies in the financial and medical fields, and enhancing user experience and the authority of information delivery.

[0086] Please refer to Figure 8 As a response to the above Figure 2 The implementation of the method shown in this application provides an embodiment of an audio-driven portrait animation generation device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0087] like Figure 8 As shown, the audio-driven portrait animation generation device of this embodiment includes: an initial information acquisition module 61, a feature extraction module 62, a latent representation sequence generation module 63, a temporal alignment processing module 64, and a portrait animation video generation module 65, wherein: The initial information acquisition module 61 is used to acquire a reference image, an audio sequence, and a text prompt sequence, wherein the text prompt sequence is used to guide the semantic information text data in the portrait animation generation; The feature extraction module 62 is used to extract features from the reference image, the audio sequence, and the text prompt sequence respectively, to obtain reference image features, audio features, and text feature vectors, and to generate text control parameters based on the text feature vectors; The latent representation sequence generation module 63 is used to generate a low-resolution latent representation sequence frame by frame using an incremental generation method based on the reference image features, the audio features and the text control parameters. The temporal alignment processing module 64 is used to input the low-resolution latent representation sequence into the high-resolution enhancement network for temporal alignment processing to generate a high-resolution latent representation sequence. The portrait animation video generation module 65 is used to decode the high-resolution latent representation sequence to generate a high-resolution portrait animation video.

[0088] Furthermore, the feature extraction module 62 includes: A reference image feature extraction unit is used to extract multi-scale visual features from the reference image through an image encoder to obtain the reference image features. An audio feature extraction unit is used to extract the audio embedding of each frame in the audio sequence through an audio encoder to generate the audio features; The text feature vector generation unit is used to convert the text prompt sequence into the text feature vector through a text encoder, and to generate the text control parameters frame by frame based on the text feature vector through a multilayer perceptron.

[0089] Furthermore, the latent representation sequence generation module 63 includes: The latent representation acquisition unit is used to acquire the latent representation of the preceding generated frame for each current frame to be generated, wherein the current frame is the target frame being generated at the current time in the incremental generation method, and the preceding generated frame is the frame before the current frame. An enhanced motion condition generation unit is used to perform anti-appearance contamination enhancement processing on the latent representation of the preceding generated frame to obtain enhanced motion conditions; The low-resolution latent representation sequence generation unit is used to generate the low-resolution latent representation sequence based on the enhanced motion conditions, the reference image features, the currently corresponding audio features, and the currently corresponding text control parameters.

[0090] Furthermore, the enhanced motion condition generation unit includes: The latent representation segmentation subunit is used to segment the latent representation of the preceding generated frame into multiple image blocks, and randomly set some of the image blocks to zero according to a preset probability. A Gaussian noise addition subunit is used to add Gaussian noise to the latent representation after block discarding to obtain the enhanced motion conditions.

[0091] Furthermore, the text control parameters include scaling parameters and offset parameters; the low-resolution latent representation sequence generation unit includes: The noise latent variable generation subunit concatenates the enhanced motion conditions with the noise latent variables of the current denoising step in the channel dimension to obtain noise latent variables that fuse motion information. An intermediate feature map generation subunit is used to input the noise latent variables of the fused motion information into a denoising network. Through the cross-attention layer of the denoising network, the reference image features and the currently corresponding audio features are used as conditions, and feature injection is performed on the noise latent variables of the fused motion information based on the conditions to obtain an intermediate feature map with fused multimodal conditions. The modulated feature map generation subunit is used to perform adaptive layer normalization modulation on the intermediate feature map using the currently corresponding scaling parameter and offset parameter to obtain the modulated feature map, so as to realize the control of facial expressions and / or postures. A new latent variable generation subunit is used to denoise the modulated feature map to obtain new latent variables output in the current denoising step; The iterative subunit is used to take the latent variable output by the new current denoising step as the noise latent variable of the next denoising step, and repeat the denoising step iteration process until the preset number of denoising steps is reached to generate the low-resolution latent representation sequence.

[0092] Furthermore, the high-resolution enhancement network is composed of multiple spatiotemporal Transformer blocks stacked together, each spatiotemporal Transformer block including a spatial self-attention layer and a temporal alignment layer; the temporal alignment processing module 64 includes: An attention calculation unit is used to independently perform spatial dimension self-attention calculation on each frame latent representation in the low-resolution latent representation sequence in the spatial self-attention layer to obtain an intermediate latent representation sequence with enhanced spatial details. A smoothing unit is used to input the spatial detail-enhanced intermediate latent representation sequence into the temporal alignment layer. In the temporal alignment layer, attention calculation and smoothing are performed on the intermediate latent representation sequence along the time dimension for features at the same spatial location to obtain the temporally coherent high-resolution latent representation sequence.

[0093] Furthermore, the portrait animation video generation module 65 includes: The decoding unit is used to input each frame of latent representation in the high-resolution latent representation sequence into a pre-trained decoder for upsampling and decoding into the high-resolution portrait animation video.

[0094] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been obtained by authorized entities (who have known and consented) or fully authorized by all parties through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0095] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 9 , Figure 9 This is a basic structural block diagram of the computer device in this embodiment.

[0096] Computer device 7 includes a memory 71, a processor 72, and a network interface 73 that are interconnected via a system bus. It should be noted that... Figure 9Only a computer device 7 with three components—memory 71, processor 72, and network interface 73—is shown. It should be understood that implementing all shown components is not required; more or fewer components may be implemented alternatively. Those skilled in the art will understand that this computer device is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), and embedded devices.

[0097] Computer devices can include desktop computers, laptops, handheld computers, and cloud servers. These devices allow for human-computer interaction with users through methods such as keyboards, mice, remote controls, touchpads, or voice-activated devices.

[0098] The memory 71 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 71 may be an internal storage unit of the computer device 7, such as the hard disk or memory of the computer device 7. In other embodiments, the memory 71 may also be an external storage device of the computer device 7, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 7. Of course, the memory 71 may include both internal storage units and external storage devices of the computer device 7. In this embodiment, the memory 71 is typically used to store the operating system and various application software installed on the computer device 7, such as program code for an audio-driven portrait animation generation method. In addition, the memory 71 may also be used to temporarily store various types of data that have been output or will be output.

[0099] In some embodiments, processor 72 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. This processor 72 is typically used to control the overall operation of the computer device 7. In this embodiment, processor 72 is used to run program code stored in memory 71 or process data, for example, to run the program code of the above-described audio-driven portrait animation generation method to implement various embodiments of the audio-driven portrait animation generation method.

[0100] The network interface 73 may include a wireless network interface or a wired network interface, which is typically used to establish a communication connection between the computer device 7 and other electronic devices.

[0101] This application also provides another embodiment, namely, providing a computer-readable storage medium storing a computer program that can be executed by at least one processor to cause the at least one processor to perform the steps of the audio-driven portrait animation generation method described above.

[0102] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of the various embodiments of this application.

[0103] Obviously, the embodiments described above are merely some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the scope of this application. This application can be implemented in many different forms; rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of protection of this application.

Claims

1. A method for generating portrait animation based on audio-driven methods, characterized in that, include: Acquire a reference image, an audio sequence, and a text prompt sequence, wherein the text prompt sequence is used to guide the semantic information in the portrait animation generation. Feature extraction is performed on the reference image, the audio sequence, and the text prompt sequence respectively to obtain reference image features, audio features, and text feature vectors, and text control parameters are generated based on the text feature vectors; Based on the reference image features, the audio features, and the text control parameters, a low-resolution latent representation sequence is generated frame by frame using an incremental generation method; The low-resolution latent representation sequence is input into a high-resolution enhancement network for time-alignment processing to generate a high-resolution latent representation sequence. The high-resolution latent representation sequence is decoded to generate a high-resolution portrait animation video.

2. The audio-driven portrait animation generation method as described in claim 1, characterized in that, The step involves extracting features from the reference image, the audio sequence, and the text prompt sequence to obtain reference image features, audio features, and text feature vectors, and generating text control parameters based on the text feature vectors, including: The reference image features are obtained by extracting multi-scale visual features from the reference image using an image encoder. The audio features are generated by extracting the audio embedding of each frame from the audio sequence using an audio encoder. The text prompt sequence is converted into the text feature vector by a text encoder, and the text control parameters are generated frame by frame by a multilayer perceptron based on the text feature vector.

3. The audio-driven portrait animation generation method as described in claim 1, characterized in that, The step of generating a low-resolution latent representation sequence frame-by-frame using an incremental generation method based on the reference image features, the audio features, and the text control parameters includes: For each current frame to be generated, obtain the latent representation of the preceding generated frame, wherein the current frame is the target frame being generated at the current moment in the incremental generation method, and the preceding generated frame is the frame before the current frame; The latent representation of the preceding generated frame is subjected to anti-appearance contamination enhancement processing to obtain enhanced motion conditions; The low-resolution latent representation sequence is generated based on the enhanced motion conditions, the reference image features, the currently corresponding audio features, and the currently corresponding text control parameters.

4. The audio-driven portrait animation generation method as described in claim 3, characterized in that, The enhancement process for the latent representation of the preceding generated frame to resist appearance contamination, resulting in enhanced motion conditions, includes: The latent representation of the preceding generated frame is divided into multiple image blocks, and some of the image blocks are randomly set to zero according to a preset probability. Gaussian noise is added to the latent representation after block discarding to obtain the enhanced motion conditions.

5. The audio-driven portrait animation generation method as described in claim 3, characterized in that, The text control parameters include scaling parameters and offset parameters; generating the low-resolution latent representation sequence based on the enhanced motion conditions, the reference image features, the currently corresponding audio features, and the currently corresponding text control parameters includes: The enhanced motion conditions are concatenated with the noise latent variables of the current denoising step in the channel dimension to obtain the noise latent variables that fuse motion information. The noise latent variables of the fused motion information are input into a denoising network. Through the cross-attention layer of the denoising network, the reference image features and the corresponding audio features are used as conditions. Based on the conditions, the noise latent variables of the fused motion information are injected with features to obtain an intermediate feature map with fused multimodal conditions. By using the scaling parameter and the offset parameter, an adaptive layer normalization modulation is performed on the intermediate feature map to obtain a modulated feature map, thereby enabling control of facial expressions and / or postures. The modulated feature map is denoised to obtain new latent variables output in the current denoising step; The latent variable output by the new current denoising step is used as the noise latent variable for the next denoising step, and the denoising step iteration is repeated until the preset number of denoising steps is reached to generate the low-resolution latent representation sequence.

6. The audio-driven portrait animation generation method as described in any one of claims 1 to 5, characterized in that, The high-resolution enhancement network is composed of multiple spatiotemporal Transformer blocks stacked together, each of which includes a spatial self-attention layer and a temporal alignment layer. The step of inputting the low-resolution latent representation sequence into a high-resolution enhancement network for time-series alignment processing to generate a high-resolution latent representation sequence includes: In the spatial self-attention layer, spatial dimension self-attention is independently calculated for each frame latent representation in the low-resolution latent representation sequence to obtain an intermediate latent representation sequence with enhanced spatial details. The spatial detail-enhanced intermediate latent representation sequence is input into the temporal alignment layer. In the temporal alignment layer, attention calculation and smoothing are performed on the features at the same spatial location along the time dimension of the intermediate latent representation sequence to obtain the temporally coherent high-resolution latent representation sequence.

7. The audio-driven portrait animation generation method as described in any one of claims 1 to 5, characterized in that, Decoding the high-resolution latent representation sequence to generate a high-resolution portrait animation video includes: Each frame of the latent representation in the high-resolution latent representation sequence is input into a pre-trained decoder for upsampling and decoding into the high-resolution portrait animation video.

8. An audio-driven portrait animation generation device, characterized in that, include: The initial information acquisition module is used to acquire reference images, audio sequences, and text prompt sequences, wherein the text prompt sequences are used to guide the semantic information text data in the portrait animation generation; The feature extraction module is used to extract features from the reference image, the audio sequence, and the text prompt sequence respectively, to obtain reference image features, audio features, and text feature vectors, and to generate text control parameters based on the text feature vectors; The latent representation sequence generation module is used to generate a low-resolution latent representation sequence frame by frame using an incremental generation method based on the reference image features, the audio features, and the text control parameters. The temporal alignment processing module is used to input the low-resolution latent representation sequence into the high-resolution enhancement network for temporal alignment processing to generate a high-resolution latent representation sequence. A portrait animation video generation module is used to decode the high-resolution latent representation sequence to generate a high-resolution portrait animation video.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the audio-driven portrait animation generation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the audio-driven portrait animation generation method as described in any one of claims 1 to 7.

Images