Portrait animation generation method, device and system based on audio driving and medium

Abstract

The invention provides a portrait animation generation method, device and system based on audio driving and a medium. The method comprises the following steps: acquiring a figure picture and animation voice data; inputting the character picture and the animation voice data into an animation generation model to obtain a portrait animation video output by the animation generation model; wherein the animation generation model is obtained by training according to a character picture sample and a motion label and a video sample corresponding to the character picture sample; and the animation generation model is used for extracting picture features according to an input figure picture, extracting audio features according to input animation voice data, performing stream matching in combination with an emotion tag predicted based on the audio features, predicting a corresponding video frame based on a potential feature sequence obtained based on a stream matching result, and obtaining a portrait animation video. According to the method, the video can be quickly generated, and natural emotion expression, consistent time sequence and stable generation of the video are ensured.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to an audio-driven portrait animation generation method, apparatus, system, and medium. Background Technology

[0002] In recent years, audio-driven portrait animation technology has attracted much attention due to its potential in virtual avatar applications. Audio-driven portrait animation technology aims to generate natural speaking movements from audio signals, including precise lip synchronization, head movements, and facial expressions.

[0003] Currently, audio-driven portrait animation technology mainly achieves generative modeling through progressive denoising using diffusion models. However, due to its iterative sampling characteristics, diffusion models are prone to insufficient temporal continuity and low sampling efficiency, resulting in low portrait animation generation efficiency. Furthermore, they heavily rely on prior facial information such as bounding boxes, 2D key points and skeletons, or 3D meshes, and their strong spatial bias greatly limits the diversity and realism of head movements. Summary of the Invention

[0004] This invention provides an audio-driven portrait animation generation method, apparatus, system, and medium to address the shortcomings of low sampling efficiency in existing technologies, which leads to low portrait animation generation efficiency. It enables rapid video generation and ensures natural emotional expression, consistent timing, and stable generation of the video.

[0005] This invention provides an audio-driven portrait animation generation method, comprising: acquiring portrait images and animated audio data; inputting the portrait images and animated audio data into an animation generation model to obtain a portrait animation video output by the animation generation model; wherein, the animation generation model is trained based on portrait image samples and corresponding motion tags and video samples; the animation generation model is used to extract image features based on the input portrait images, extract audio features based on the input animated audio data, and perform stream matching by combining emotion tags predicted based on audio features, and predict corresponding video frames based on the latent feature sequence obtained from the stream matching results to obtain the portrait animation video.

[0006] According to the present invention, an audio-driven portrait animation generation method is provided. The animation generation model includes: a motion autoencoder, which encodes the input image of a person to obtain image features, and decomposes the image features to obtain identity features and action features; an audio encoder, which encodes the input animated speech data to obtain audio features; a sentiment predictor, which predicts the sentiment of the speech features to obtain sentiment labels; a condition generator, which generates driving conditions based on audio features, sentiment labels, and action features; a vector field predictor, which obtains a motion latent variable sequence based on action features and driving conditions, using layer normalization, gating, and masked self-attention mechanisms, and using the target vector field obtained by the model in advance; and a decoder, which adds the identity features to the motion latent variable sequence and decodes it to obtain the portrait animation video.

[0007] According to the present invention, a portrait animation generation method based on audio-driven processing is provided. The driving conditions include driving conditions corresponding to each stream time step. Layer normalization, gating, and mask self-attention mechanisms are used, and a motion latent variable sequence is obtained using the target vector field obtained through prior model training. This includes: SA1, obtaining modulation coefficients for each stream time step using linear transformation based on the driving conditions, where the modulation coefficients include a first coefficient, a second coefficient, and a third coefficient; SA2, using motion features as features of the current stream time step, and performing adaptive layer normalization on the features of the current stream time step using the second and third coefficients of the corresponding stream time step to obtain the layer-normalized features of the current stream time step; SA3, based on the current stream time... The first coefficient of the inter-step process is used to perform gating processing on the layer normalized features of the current flow time step to obtain the gating processing features of the current flow time step; SA4, based on the gating processing features of the current flow time step, combined with the mask self-attention mechanism, captures the context information of the gating processing features of adjacent flow time steps to obtain the latent features corresponding to the current flow time step; SA5, based on the latent features corresponding to the current flow time step, uses the target vector field obtained by prior training to predict the motion latent variables of the next flow time step; SA6, uses the predicted motion latent variables of the next flow time step as the features of the current flow time step, and repeats steps SA2-SA6 until the motion latent variables corresponding to all flow time steps are obtained, resulting in a sequence of motion latent variables.

[0008] According to the present invention, a portrait animation generation method based on audio-driven transformation is provided. Based on the driving conditions, a linear transformation is used to obtain the modulation coefficients of each stream time step. The modulation coefficients include a first coefficient, a second coefficient, and a third coefficient. The method includes: for the driving conditions corresponding to each stream time step, using the target linear transformation layer ToScaleShift, mapping the driving conditions corresponding to each stream time step to a high-dimensional space to obtain the modulation coefficients; and dividing the modulation coefficients according to the mapping dimension order to obtain the first coefficient, the second coefficient, and the third coefficient.

[0009] According to the present invention, a portrait animation generation method based on audio-driven features generates driving conditions based on audio features, emotion tags, and action features, including: concatenating audio features, emotion tags, and action features to obtain concatenated features; using sinusoidal position embedding to obtain a streaming time step embedding vector based on a streaming time step determined prior to a preset number of video frames and the duration of the animation audio data; and adding the streaming time step embedding vector to the concatenated features to obtain the driving conditions.

[0010] According to the present invention, a portrait animation generation method based on audio-driven processing includes the following steps before inputting portrait images and animated audio data into the animation generation model: SB1, acquiring portrait image samples and corresponding motion labels and video samples; wherein the motion labels are used to characterize the initial motion state corresponding to the portrait image samples; SB2, constructing a ground truth vector field based on the motion labels and video samples; wherein the ground truth vector field is used to define the motion path from the initial motion state to the target motion state determined based on the initial motion state and video samples; SB3, inputting portrait image samples into the model to be trained to predict the vector field, and determining the predicted video frames corresponding to each streaming time step based on the predicted vector field; SB4, obtaining an objective function based on the predicted vector field and the ground truth vector field, adjusting the model parameters based on the objective function value, and repeating steps SB3-SB4 until the objective function converges, taking the finally predicted vector field as the target vector field, and ending the training.

[0011] According to the present invention, an audio-driven portrait animation generation method is provided, which obtains an objective function based on a predicted vector field and a real vector field, including: determining the predicted motion state corresponding to each flow time step based on the predicted vector field, and determining the real motion state corresponding to each flow time step based on the real vector field; constructing a flow matching loss function based on the predicted motion state and the real motion state corresponding to each flow time step; determining the motion prediction difference between two adjacent flow time steps based on the predicted vector field, and determining the real motion difference between two adjacent flow time steps based on the real vector field; constructing a velocity loss function based on the motion prediction difference and the real motion difference between two adjacent flow time steps; and obtaining the objective function based on the flow matching loss function and the velocity loss function.

[0012] This invention also provides an audio-driven portrait animation generation device, comprising: a data acquisition module for acquiring person images and animated audio data; and an animation generation module for inputting the person images and animated audio data into an animation generation model to obtain a portrait animation video output by the animation generation model. The animation generation model is trained based on person image samples, corresponding motion tags, and video samples. The animation generation model extracts image features from the input person images, extracts audio features from the input animated audio data, and combines this with emotion tag prediction based on the audio features for stream matching. It then predicts corresponding video frames based on the latent feature sequence obtained from the stream matching results to obtain the portrait animation video.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the audio-driven portrait animation generation method described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the audio-driven portrait animation generation method as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the audio-driven portrait animation generation method described above.

[0016] This invention provides an audio-driven portrait animation generation method, apparatus, system, and medium. It acquires a person's image to facilitate the subsequent generation of a portrait animation matching that person. Combined with the animation's audio data, it ensures accurate and natural video content, facilitating multimodal processing by the subsequent model and improving generation results. The acquired person image and animation audio data are input into the animation generation model, which simultaneously processes both data to enhance the animation's naturalness and expressiveness. It also predicts sentiment tags, making the animated character's expressions and movements more consistent with the emotional tone of the audio content. Stream matching of sentiment tags, image features, and audio features ensures a high degree of match between speech rhythm and facial movements, improving temporal consistency. Finally, it generates a video based on latent feature sequences to maintain animation continuity and stability. This allows for rapid processing of input data and video generation, ensuring natural emotional expression, temporal consistency, and stable generation. It is suitable for large-scale, real-time generation scenarios, eliminating the need for manual frame-by-frame production and saving significant time and labor costs. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the audio-driven portrait animation generation method provided by the present invention. Figure 2 This is a schematic diagram of the architecture of the animation generation model provided by the present invention; Figure 3 This is a schematic diagram of the audio-driven portrait animation generation device provided by the present invention; Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0020] Figure 1 This is a flowchart illustrating the audio-driven portrait animation generation method provided by the present invention, as shown below. Figure 1 As shown, the method includes: S11, acquire character images and animation voice data; S12, the person image and animated speech data are input into the animation generation model to obtain the portrait animation video output by the animation generation model; wherein, the animation generation model is trained based on the person image sample and the motion label corresponding to the person image sample and the video sample; the animation generation model is used to extract image features based on the input person image, extract audio features based on the input animated speech data, and combine the predicted sentiment label based on the audio features for stream matching, and predict the corresponding video frame based on the latent feature sequence obtained based on the stream matching result to obtain the portrait animation video.

[0021] It should be noted that the step number "S1N" in this manual does not represent the order of the audio-driven portrait animation generation methods. The following details will explain further. Figure 2 This invention describes an audio-driven portrait animation generation method.

[0022] Step S11: Obtain character images and animated voice data.

[0023] It should be added that the animated audio data is used to represent the audio content in the portrait animation video to be generated. Additionally, the portrait image can be selected based on the actual design. In this context, R represents the real number field, 3 represents the number of channels (usually RGB three channels), H represents the image height, and W represents the image width.

[0024] Step S12: Input the person image and animated speech data into the animation generation model to obtain the portrait animation video output by the animation generation model; wherein, the animation generation model is trained based on the person image sample and the motion label and video sample corresponding to the person image sample; the animation generation model is used to extract image features based on the input person image, extract audio features based on the input animated speech data, and combine the predicted sentiment label based on the audio features for stream matching, and predict the corresponding video frame based on the latent feature sequence obtained based on the stream matching result to obtain the portrait animation video.

[0025] In this embodiment, reference Figure 2 The animation generation model includes: a motion autoencoder, which encodes the input image of a person to obtain image features, and decomposes the image features to obtain identity features and action features; an audio encoder, which encodes the input animated speech data to obtain audio features; a sentiment predictor, which predicts the sentiment of the speech features to obtain sentiment labels; a condition generator, which generates driving conditions based on audio features, sentiment labels, and action features; a vector field predictor, which obtains a sequence of motion latent variables based on action features and driving conditions, using layer normalization, gating, and masked self-attention mechanisms, and utilizing the target vector field obtained from the model's prior training; and a decoder, which adds identity features to the sequence of motion latent variables and decodes it to obtain the portrait animation video.

[0026] It should be added that motion autoencoders can employ latent image animators (LIA). Since the expressiveness of generated motion and image fidelity are determined by the motion latent space, the original LIA architecture is extended to synthesize from 256... 2 up to 512 2Higher resolution. The training of motion autoencoders can be achieved by introducing facial component perception loss. This is achieved by extracting image features and high-level features of people images through a pre-trained discriminative network (such as VGG) and minimizing the distance between these feature representations to generate images that are perceptually more similar. This allows LIA to be trained to reconstruct driving images from source images sampled from the same video clip, forcing the latent encoding to include implicit motion. This ensures that temporally adjacent and distant motions can be captured, significantly improving image fidelity (e.g., details such as teeth and eyes) while also improving fine motion (e.g., the movement of eyeballs and eyebrows).

[0027] Accordingly, the input images of people are encoded to obtain image features, and these image features are decomposed to obtain identity features and action features. This includes: encoding the images of people into a latent space using LIA to obtain latent vectors, i.e., image features; and using a motion basis V independent of the images of people to explicitly decompose the image features to obtain identity features and action features. This transfers the image features from a pixel-based latent space to a learned motion latent space, achieving temporally consistent motion and reducing computational load.

[0028] It should be noted that explicit decomposition is expressed as: in, Indicates image features, , Let represent a d-dimensional real vector space, i.e., the latent space; Indicates identity characteristics, ; Indicates the characteristics of motion.

[0029] Furthermore, the motion characteristics are represented as follows: in, The intensity of motion features is represented by motion coefficients from the person image, which span a learned motion orthogonal basis independent of the person image. M represents the number of orthogonal motions corresponding to a single motion.

[0030] Furthermore, when encoding animation audio data using an audio encoder, a sliding window of preset length L is used to select and encode the animation audio data. The step size L-L' of the sliding window is less than the length L of the sliding window each time, ensuring that the later selected audio data includes the last L' frame of the previously selected audio data, thus obtaining the corresponding audio features. , Dimensions representing audio features.

[0031] Furthermore, during speech, humans naturally express emotions through sound, and these emotions influence speaking gestures. For example, a sad speaker might be more inclined to shake their head and avoid eye contact. These nonverbal cues derived from emotion are crucial to the naturalness of the speaker's demeanor. Therefore, by using a pre-trained emotion predictor to predict emotions from speech features, and predicting the softmax probability of preset emotions based on these features, emotion labels can be obtained. These preset emotions can be set according to actual design needs, such as anger, disgust, fear, happiness, neutrality, sadness, and surprise. Without further limitations, we can extract emotional tags from audio features to avoid manual annotation.

[0032] Specifically, the condition generator can employ a linear layer called ToCondition. Based on audio features, emotion tags, and action features, it generates driving conditions, including: [the following is a list of audio features and their relationships to the previous sentence]. Emotional tags and action characteristics The concatenation is performed to obtain the concatenation features; based on the stream time step determined prior to the preset number of video frames and the duration of the animation audio data, sinusoidal position embedding is used to obtain the stream time step embedding vector; the stream time step embedding vector is added to the concatenation features to obtain the driving conditions.

[0033] It should be noted that since people do not always speak with a single, clear emotion, determining the emotion solely from the audio is often ambiguous. Therefore, in addition to emotion labels, motion features and audio features are combined to generate driving conditions. This avoids simply introducing voice-driven emotions, which would make emotion perception and action generation more challenging. This supports voice-driven emotion enhancement and ensures that it is naturally integrated into expressive actions.

[0034] Additionally, the vector field predictor can be used for Flow Matching Transformer (FMT). Built using a DiT architecture, it decouples frame-by-frame conditions from the temporal axis attention mechanism to model temporally consistent motion latent variables. In DiT, different semantic tags are modulated by a single diffuse temporal step embedding and class embedding through Adaptive Layer Normalization (AdaLN), frame-by-frame AdaLN and frame-by-frame gating, and then their temporal relationships are combined through a masked self-attention layer to establish constant-time motion dependencies. This enforces motion smoothing constraints in the latent space, reducing flicker. The neighboring frames that the masked self-attention layer focuses on can be set according to actual design requirements, such as 2⋅T neighboring frames; no further limitations are made here.

[0035] Furthermore, the driving conditions include the driving conditions corresponding to each flow time step. Layer normalization, gating, and masked self-attention mechanisms are used, and the target vector field obtained from prior model training is used to obtain the motion latent variable sequence, including: SA1, based on the driving conditions, using linear transformation to obtain the modulation coefficients of each flow time step, the modulation coefficients include the first coefficient, the second coefficient, and the third coefficient; SA2, using the action features as the features of the current flow time step, using the second and third coefficients of the corresponding flow time step to perform adaptive layer normalization on the features of the current flow time step, obtaining the layer-normalized features of the current flow time step; SA3, based on the first coefficient of the current flow time step, the motion latent variable sequence is obtained. The normalized features of each streaming time step are gating-processed to obtain the gating features of the current streaming time step; SA4, based on the gating features of the current streaming time step, combined with a masked self-attention mechanism, the contextual information of the gating features of adjacent streaming time steps is captured to obtain the latent features corresponding to the current streaming time step; SA5, based on the latent features corresponding to the current streaming time step, the motion latent variables of the next streaming time step are predicted using the target vector field obtained through prior training; SA6, the predicted motion latent variables of the next streaming time step are used as the features of the current streaming time step, and steps SA2-SA6 are executed iteratively until the motion latent variables corresponding to all streaming time steps are obtained, resulting in a sequence of motion latent variables.

[0036] It should be noted that the layer normalization feature , represented as: in, This represents the second coefficient corresponding to the l-th driving condition at the t-th flow time step; This represents the third coefficient corresponding to the l-th driving condition at the t-th flow time step; LN represents the l-th input of each operation within time step t∈[0,1] of the corresponding stream; LN represents layer normalization; h represents the hidden dimension.

[0037] Gating processing features , represented as: in, This represents the first coefficient of the l-th driving condition corresponding to the t-th flow time step.

[0038] Furthermore, based on the driving conditions, a linear transformation is used to obtain the modulation coefficients for each stream time step. The modulation coefficients include a first coefficient, a second coefficient, and a third coefficient. This process involves: using the target linear transformation layer ToScaleShift to map the driving conditions corresponding to each stream time step to a high-dimensional space to obtain the modulation coefficients; and dividing the modulation coefficients according to the mapping dimension to obtain the first coefficient, the second coefficient, and the third coefficient.

[0039] It should be added that the modulation coefficient is expressed as: in, Indicates the modulation coefficient; This indicates that the t-th flow time step corresponds to the l-th driving condition. The ToScaleShift algorithm maps the h-dimensional driving conditions to a 3h-dimensional output and then splits it into independent parameter groups according to the dimensions, namely the first coefficient, the second coefficient, and the third coefficient. Thus, the first, second, and third coefficients can be used for different modulations, namely AdaLN and gating, as described above.

[0040] Furthermore, after obtaining the motion latent variables corresponding to all streaming time steps, ordinary differential equations (ODEs) can be used to solve them. The motion latent variables can be calculated by numerical integration, and the identity features can be added to the calculated motion latent variables. In this way, the motion decoder can be used to decode them into video frames, avoiding the distribution offset problem of the pre-trained encoder.

[0041] In one alternative embodiment, before inputting the character images and animated voice data into the animation generation model, the following steps are included: SB1, obtain the image sample of the person and the corresponding motion label and video sample of the image sample; whereby the motion label is used to characterize the initial motion state corresponding to the image sample of the person; SB2, constructs a real vector field based on motion tags and video samples; whereby the real vector field is used to define the motion path from the initial motion state to the target motion state determined based on the initial motion state and video samples; SB3 inputs human image samples into the model to be trained to predict the vector field, and determines the predicted video frame corresponding to each stream time step based on the predicted vector field. SB4: Based on the predicted vector field and the true vector field, obtain the objective function. Adjust the model parameters based on the objective function value and repeat steps SB3-SB4 until the objective function converges. Use the final predicted vector field as the objective vector field and end the training.

[0042] It should be noted that the method by which the model to be trained obtains the corresponding motion latent variable sequence based on the input human image samples and the predicted vector field can be referred to the above, and no further limitations are made here.

[0043] It should be noted that during model training, the driving conditions of the Conditional Flow Vector (CFV) need to be expanded to incremental CFV to adjust audio and emotion separately. This avoids the situation where simply introducing speech-driven emotion makes emotion-aware action generation more challenging. The corresponding motion latent variables are represented as follows: in, This represents the adjusted latent motion variables; The vector field predicted by the model represents the motion latent variables obtained from incremental CFV prediction. This indicates the absence of driving conditions for x and y, where x and y refer to the audio features in the above formula. and emotional tags ; and These represent the guidance scales for audio and emotion, respectively. Additionally, during training, motion characteristics are also considered. Emotional tags and audio features Dropout with a probability of 0.1 was applied for the aforementioned incremental CFV. Furthermore, to achieve a smooth transition within the initial sliding window, the prior audio features and vector fields were... Dropout with a probability of 0.5 was applied.

[0044] Specifically, based on the predicted vector field and the true vector field, the objective function is obtained, including: determining the predicted motion state corresponding to each flow time step based on the predicted vector field, and determining the true motion state corresponding to each flow time step based on the true vector field; constructing a flow matching loss function based on the predicted motion state and the true motion state corresponding to each flow time step; determining the motion prediction difference between two adjacent flow time steps based on the predicted vector field, and determining the actual motion difference between two adjacent flow time steps based on the true vector field; constructing a velocity loss function based on the motion prediction difference and the actual motion difference between two adjacent flow time steps; and obtaining the objective function based on the flow matching loss function and the velocity loss function.

[0045] It should be added that by jointly optimizing the stream matching loss function and the velocity loss function, the motion of the generated portrait video is made more natural and consistent with the audio-driven logic. The objective function is expressed as: in, Represents model parameters The corresponding objective function; and These represent the balance coefficients; This represents the flow matching loss function; Represents the velocity loss function; Indicates in model parameters The predicted vector field under the given conditions; This represents the motion characteristics after adding noise; Indicates the driving condition; Indicates the addition of random noise The initial motion state can be determined based on the motion tag. It follows a Gaussian distribution; This indicates that based on motion tags and video samples ( The constructed real vector field is used to represent the representation of the vector field from... arrive The change in motion, , This represents the latent variable driving the motion, i.e., the target motion state; This indicates the difference in motion prediction between two adjacent frames; This indicates the actual difference in motion between two adjacent frames.

[0046] It should be added that the real vector field Essentially, it is a mathematical representation of motion changes. It is constructed by selecting the corresponding source-driven motion and the target motion latent variables based on the motion labels and corresponding audio samples. It defines the ideal motion path from the initial state to the target state and uses the real vector field to guide the model to predict the vector field. Learn how to smoothly transition from the initial state x0 to the target motion. ,make As close as possible to the real vector field .

[0047] In summary, this invention acquires images of people to facilitate the generation of portrait animations that match those images. Combined with animated audio data, it ensures accurate and natural video content, facilitating multimodal processing by the subsequent model and improving generation quality. The acquired images and animated audio data are input into the animation generation model, allowing the model to process both simultaneously, enhancing the naturalness and expressiveness of the animation. It also predicts sentiment tags, making the animated character's expressions and movements more consistent with the emotional tone of the audio content. Stream matching of sentiment tags, image features, and audio features ensures a high degree of match between speech rhythm and facial movements, improving temporal consistency. Finally, it generates videos based on latent feature sequences, maintaining the animation's continuity and stability. This allows for rapid processing of input data and video generation, ensuring natural emotional expression, temporal consistency, and stable generation. It is suitable for large-scale, real-time generation scenarios, eliminating the need for manual frame-by-frame production and saving significant time and manpower costs.

[0048] The following describes the audio-driven portrait animation generation apparatus provided by the present invention. The audio-driven portrait animation generation apparatus described below and the audio-driven portrait animation generation method described above can be referred to in correspondence.

[0049] Figure 3 A schematic diagram of an audio-driven portrait animation generation device is shown, the device comprising: Data acquisition module 31 acquires character images and animated voice data; The animation generation module 32 inputs the person image and animated audio data into the animation generation model to obtain the portrait animation video output by the animation generation model. The animation generation model is trained based on the person image samples and the corresponding motion tags and video samples. The animation generation model is used to extract image features from the input person image, extract audio features from the input animated audio data, and combine them with the predicted sentiment tags based on the audio features for stream matching. Based on the latent feature sequence obtained from the stream matching results, the corresponding video frames are predicted to obtain the portrait animation video.

[0050] It should be noted that the specific principles of the embodiments of the present invention are the same as those of the method embodiments described above. For details, please refer to the method embodiments above. More detailed explanations will not be repeated here.

[0051] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4As shown, the electronic device may include a processor 410, a communications interface 420, a memory 430, and a communication bus 440. The processor 410, communications interface 420, and memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute an audio-driven portrait animation generation method. This method includes: acquiring a person's image and animated audio data; inputting the person's image and animated audio data into an animation generation model to obtain a portrait animation video output by the animation generation model; wherein the animation generation model is trained based on person image samples and corresponding motion tags and video samples; the animation generation model is used to extract image features from the input person image, extract audio features from the input animated audio data, and combine this with emotion tag prediction based on audio features for stream matching, predicting corresponding video frames based on the latent feature sequence obtained from the stream matching result to obtain the portrait animation video.

[0052] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0053] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the audio-driven portrait animation generation method provided by the above methods. The method includes: acquiring a person image and animated audio data; inputting the person image and animated audio data into an animation generation model to obtain a portrait animation video output by the animation generation model; wherein the animation generation model is trained based on person image samples and corresponding motion tags and video samples; the animation generation model is used to extract image features based on the input person image, extract audio features based on the input animated audio data, and perform stream matching by combining emotion tag prediction based on audio features, and predict corresponding video frames based on the latent feature sequence obtained from the stream matching result to obtain the portrait animation video.

[0054] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the audio-driven portrait animation generation method provided by the above methods. The method includes: acquiring a person image and animated audio data; inputting the person image and animated audio data into an animation generation model to obtain a portrait animation video output by the animation generation model; wherein the animation generation model is trained based on person image samples and corresponding motion tags and video samples; the animation generation model is used to extract image features based on the input person image, extract audio features based on the input animated audio data, and perform stream matching by combining emotion tag prediction based on audio features, and predict corresponding video frames based on the latent feature sequence obtained from the stream matching result to obtain the portrait animation video.

[0055] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0056] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0057] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating portrait animation based on audio-driven methods, characterized in that, include: Acquire character images and animated voice data; The person image and the animated audio data are input into the animation generation model to obtain the portrait animation video output by the animation generation model; wherein, the animation generation model is trained based on the person image sample and the motion tags and video samples corresponding to the person image sample; The animation generation model is used to extract image features from the input image of a person, extract audio features from the input animation audio data, and combine the predicted sentiment tags based on the audio features to perform stream matching. Based on the latent feature sequence obtained from the stream matching results, the model predicts the corresponding video frames to obtain the portrait animation video.

2. The audio-driven portrait animation generation method according to claim 1, characterized in that, The animation generation model includes: The motion autoencoder encodes the input image of a person to obtain image features, and decomposes the image features to obtain identity features and action features. An audio encoder encodes the input animated audio data to obtain audio features; The sentiment predictor performs sentiment prediction on the speech features to obtain sentiment labels; A condition generator generates driving conditions based on the audio features, the emotion tags, and the action features; The vector field predictor, based on the action features and the driving conditions, utilizes layer normalization, gating, and masked self-attention mechanisms, and uses the target vector field obtained from the prior training of the model to obtain the sequence of motion latent variables. The decoder adds the identity features to the motion latent variable sequence and decodes it to obtain the portrait animation video.

3. The audio-driven portrait animation generation method according to claim 2, characterized in that, The driving conditions include the driving conditions corresponding to each flow time step. Layer normalization, gating, and masked self-attention mechanisms are used, and the motion latent variable sequence is obtained using the target vector field obtained from prior model training. This sequence includes: SA1, based on the driving conditions, uses linear transformation to obtain the modulation coefficients for each time step of the stream, the modulation coefficients including a first coefficient, a second coefficient, and a third coefficient; SA2, the action features are used as the features of the current flow time step, and the features of the current flow time step are adaptively normalized using the second and third coefficients of the corresponding flow time step to obtain the normalized features of the current flow time step. SA3, based on the first coefficient of the current flow time step, performs gating processing on the layer normalization feature of the current flow time step to obtain the gating processing feature of the current flow time step; SA4, based on the gating processing features of the current stream time step, combined with the mask self-attention mechanism, captures the context information of the gating processing features of adjacent stream time steps to obtain the latent features corresponding to the current stream time step; SA5, based on the latent features corresponding to the current flow time step, uses the target vector field obtained through prior training to predict the motion latent variables of the next flow time step; SA6 uses the predicted motion latent variables of the next flow time step as features of the current flow time step, and repeats steps SA2-SA6 until the motion latent variables corresponding to all flow time steps are obtained, thus obtaining a sequence of motion latent variables.

4. The audio-driven portrait animation generation method according to claim 3, characterized in that, Based on the driving conditions, a linear transformation is used to obtain the modulation coefficients for each time step of the stream. These modulation coefficients include a first coefficient, a second coefficient, and a third coefficient, comprising: For each stream time step, the driving conditions are mapped to a high-dimensional space using the target linear transform layer ToScaleShift to obtain the modulation coefficients. The modulation coefficients are divided according to the mapping dimension to obtain the first coefficient, the second coefficient, and the third coefficient.

5. The audio-driven portrait animation generation method according to claim 2, characterized in that, Based on the audio features, the emotion tags, and the action features, driving conditions are generated, including: The audio features, the emotion tags, and the action features are concatenated to obtain the concatenated features; Based on the stream time step determined prior to the preset video frame count and animation audio data duration, a stream time step embedding vector is obtained using sinusoidal position embedding. The streaming time step embedding vector is added to the splicing feature to obtain the driving condition.

6. The audio-driven portrait animation generation method according to claim 1, characterized in that, Before inputting the character image and the animated voice data into the animation generation model, the process includes: SB1, acquire a person image sample and the corresponding motion tag and video sample; wherein, the motion tag is used to characterize the initial motion state corresponding to the person image sample; SB2, construct a real vector field based on the motion tag and the video sample; wherein, the real vector field is used to define the motion path from the initial motion state to the target motion state determined based on the initial motion state and the video sample; SB3, input the image samples of the people into the model to be trained to predict the vector field, and determine the predicted video frame corresponding to each time step based on the predicted vector field; SB4: Based on the predicted vector field and the true vector field, obtain the objective function, adjust the model parameters based on the objective function value, and repeat steps SB3-SB4 until the objective function converges. Then, take the finally predicted vector field as the target vector field and end the training.

7. The audio-driven portrait animation generation method according to claim 6, characterized in that, Based on the predicted vector field and the true vector field, the objective function is obtained, including: Based on the predicted vector field, the predicted motion state corresponding to each flow time step is determined, and based on the actual vector field, the actual motion state corresponding to each flow time step is determined. Based on the predicted motion state corresponding to each flow time step and the actual motion state corresponding to each flow time step, a flow matching loss function is constructed. Based on the predicted vector field, determine the motion prediction difference between two adjacent flow time steps, and based on the actual vector field, determine the actual motion difference between two adjacent flow time steps. Based on the motion prediction difference and the actual motion difference between the two adjacent time steps, a velocity loss function is constructed. The objective function is obtained based on the flow matching loss function and the velocity loss function.

8. An audio-driven portrait animation generation device, characterized in that, include: The data acquisition module acquires character images and animated voice data; An animation generation module inputs the character image and the animated audio data into an animation generation model to obtain a portrait animation video output by the animation generation model; wherein, the animation generation model is trained based on character image samples and the motion tags and video samples corresponding to the character image samples; The animation generation model is used to extract image features from the input image of a person, extract audio features from the input animation audio data, and combine the predicted sentiment tags based on the audio features to perform stream matching. Based on the latent feature sequence obtained from the stream matching results, the model predicts the corresponding video frames to obtain the portrait animation video.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the audio-driven portrait animation generation method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the audio-driven portrait animation generation method as described in any one of claims 1 to 7.

Images