Expression driving model training method, expression driving method and corresponding device

Abstract

The embodiment of the invention discloses a training method of an expression driving model, an expression driving method and a corresponding device. According to the main technical scheme, first training data comprising a plurality of first training samples are obtained, the first training samples at least comprise audio samples, expression control data corresponding to the audio samples and expression parameter truth values, and the expression control data at least comprise time sequence control data; the time sequence control data comprises an action type sequence corresponding to at least one time window in a time sequence of the audio sample; and training an expression driving model based on the first training data: taking the audio sample and an expression control parameter corresponding to the audio sample as input of the expression driving model, and obtaining an expression parameter sequence output by the expression driving model. According to the invention, the fine control effect of the facial expression can be improved, and the expression change is richer and more vivid.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a training method for an expression-driven model, an expression-driven method, and a corresponding device. Background Art

[0002] Generating realistic speech-driven facial animation is of great significance to the fields of filmmaking and artificial intelligence. It not only requires precise lip synchronization and vivid facial expressions, but also has a wide range of applications. It can significantly enhance the visual impact of movies and play a key role in many fields such as virtual reality, games and social media.

[0003] However, existing technical solutions have limitations in the generation of voice-driven facial expressions, and it is difficult to achieve fine-grained control of expressions, which limits the emotional expressiveness and delicacy of facial expressions, and makes it difficult to generate facial expressions that meet expectations and are highly expressive. These limitations are particularly evident in scenarios that require fine emotional expression. Summary of the invention

[0004] In view of this, the present application provides a training method for an expression-driven model, an expression-driven method and a corresponding device, so as to improve the refined control effect of facial expressions and make the expression changes richer and more vivid.

[0005] This application provides the following solutions:

[0006] In a first aspect, a method for training an expression-driven model is provided, the method comprising:

[0007] Acquire first training data including a plurality of first training samples, wherein the first training samples at least include: an audio sample, expression control data corresponding to the audio sample, and a true value of an expression parameter, wherein the expression control data at least includes timing control data, and the timing control data includes an action type sequence corresponding to at least one time window in the timing of the audio sample;

[0008] The expression driven model is trained based on the first training data, wherein the training includes: taking the audio sample and the expression control parameters corresponding to the audio sample as inputs of the expression driven model, obtaining the expression parameter sequence output by the expression driven model, and updating the model parameters of the expression driven model using the loss function corresponding to the first training objective, wherein the first training objective includes minimizing the difference between the expression parameter sequence output by the expression driven model and the corresponding true value of the expression parameter.

[0009] Optionally, the expression control data further includes global control data, and the global control data includes one or any combination of an emotion category label, a speaker style label, and an emotion intensity.

[0010] Optionally, the expression driven model includes: an audio coding network, a global coding network, a temporal coding network and a diffusion network;

[0011] The audio coding network is used to encode the audio sample to obtain a feature representation sequence corresponding to the audio sample;

[0012] The global encoding network is used to encode the global control data to obtain a feature representation corresponding to the global control data;

[0013] The timing coding network is used to encode the timing control data to obtain a feature representation sequence corresponding to the timing control data;

[0014] The diffusion network is used to denoise the noise feature sequence using the feature representation sequence corresponding to the audio sample, the feature representation corresponding to the global control data and the feature representation corresponding to the timing control data, and predict the expression parameter sequence using the feature sequence obtained by the denoising process.

[0015] Optionally, the training includes a first training phase and a second training phase;

[0016] In the first training stage, the audio sample and the global control data are used as inputs of the expression drive model, a first loss function value is determined by using the difference between the expression parameter sequence output by the expression drive model and the corresponding expression parameter true value, and the model parameters of the expression drive model are updated by using the first loss function value;

[0017] In the second training stage, an adaptation module is added to at least one Transformer layer of the diffusion network, and the adaptation module is used to introduce the feature representation corresponding to the timing control data into the denoising process; the audio samples, random global control data and the timing control data are used as inputs of the expression driving model, and the value of the second loss function is determined by the difference between the expression parameter sequence output by the expression driving model for the at least one time window and the corresponding expression parameter true value, and the second loss function value is used to update only the model parameters of the adaptation module in the expression driving model.

[0018] Optionally, the global encoding network includes: a label encoding subnetwork and an intensity embedding module;

[0019] The label encoding subnetwork is used to encode the emotion category label and the speaker style label to obtain a feature representation of the emotion category label and a feature representation of the speaker style label respectively;

[0020] The intensity embedding module is used to scale the intensity embedding vector according to the emotion intensity to obtain a feature representation of the emotion intensity, and the intensity embedding vector is continuously optimized during the training process;

[0021] The feature representation of the emotion category label, the feature representation of the speaker style label and the feature representation of the emotion intensity are fused to obtain the feature representation corresponding to the global control data.

[0022] Optionally, the adaptation module includes a cross-attention layer and a zero convolution layer, the cross-attention layer performs cross-attention processing on the feature representation corresponding to the timing control data and the latent vector sequence output by the Transformer layer, and provides the latent vector sequence obtained by the cross-attention processing to the zero convolution layer for optimization, and the zero convolution layer outputs the optimized latent vector sequence to the next Transformer layer.

[0023] Optionally, the timing control data is obtained in the following manner:

[0024] Get the action description text;

[0025] An action detection module is used to predict an action type sequence corresponding to at least one time window based on the action description text.

[0026] Optionally, the action detection module is pre-trained in the following manner:

[0027] Acquire second training data including a plurality of sample pairs, wherein the sample pairs include two action description texts, and the second training data also includes action type sequence samples corresponding to each action description sample;

[0028] The action detection module is trained using the second training data, including: inputting the action description text into the action detection module, the action detection module extracting feature representations from the action description text, and predicting the action type sequence using the extracted feature representations; and updating the parameters of the action detection module using the loss function value corresponding to the second training objective, wherein the second training objective includes: minimizing the distance between the feature representations of two action description texts in a positive sample pair, maximizing the distance between the feature representations of two action description texts in a negative sample pair, and minimizing the difference between the predicted action type sequence and the corresponding action type sequence sample.

[0029] In a second aspect, an expression driving method is provided, the method comprising:

[0030] Acquire audio data and expression control data, wherein the expression control data at least includes timing control data, and the timing control data includes an action type sequence corresponding to at least one time window in the timing of the audio data;

[0031] Inputting the audio data and the expression control data into an expression driving model, obtaining an expression parameter sequence predicted by the expression driving model using the audio data and the expression control data, wherein the expression parameter sequence corresponds to the time sequence of the audio data;

[0032] Based on the expression parameter sequence, the facial expression of the virtual character is controlled.

[0033] Optionally, the expression control data further includes global control data, and the global control data includes one or any combination of an emotion category label, a speaker style label, and an emotion intensity.

[0034] Optionally, the audio coding network is used to encode the audio data to obtain a feature representation sequence corresponding to the audio data;

[0035] The global encoding network is used to encode the global control data to obtain a feature representation corresponding to the global control data;

[0036] The timing coding network is used to encode the timing control data to obtain a feature representation sequence corresponding to the timing control data;

[0037] The diffusion network is used to denoise the noise feature sequence using the feature representation sequence corresponding to the audio data, the feature representation corresponding to the global control data and the feature representation corresponding to the timing control data, and predict the expression parameter sequence using the feature sequence obtained by the denoising process.

[0038] Optionally, the global encoding network includes: a label encoding subnetwork and an intensity embedding module;

[0039] The label encoding subnetwork is used to encode the emotion category label and the speaker style label to obtain a feature representation of the emotion category label and a feature representation of the speaker style label respectively;

[0040] The intensity embedding module is used to scale the intensity embedding vector according to the emotion intensity to obtain a feature representation of the emotion intensity, and the intensity embedding vector is continuously optimized during the training process;

[0041] The feature representation of the emotion category label, the feature representation of the speaker style label and the feature representation of the emotion intensity are fused to obtain the feature representation corresponding to the global control data.

[0042] Optionally, an adaptation module is attached to at least one Transformer layer of the diffusion network, and the adaptation module is used to introduce the feature representation corresponding to the timing control data into the denoising process.

[0043] Optionally, the adaptation module includes a cross-attention layer and a zero convolution layer, the cross-attention layer performs cross-attention processing on the feature representation corresponding to the timing control data and the latent vector sequence output by the Transformer layer, and provides the latent vector sequence obtained by the cross-attention processing to the zero convolution layer for optimization, and the zero convolution layer outputs the optimized latent vector sequence to the next Transformer layer.

[0044] Optionally, the method further comprises:

[0045] The adaptation module performs mask processing on a time sequence portion outside the at least one time window in the input latent vector sequence.

[0046] Optionally, acquiring the timing control data includes:

[0047] Get the action description text;

[0048] An action detection module is used to predict an action type sequence corresponding to at least one time window based on the action description text.

[0049] In a third aspect, a training device for an expression-driven model is provided, the device comprising:

[0050] A sample acquisition unit is configured to acquire first training data including a plurality of first training samples, wherein the first training samples at least include: an audio sample, expression control data corresponding to the audio sample, and a true value of an expression parameter, wherein the expression control data at least includes timing control data, and the timing control data includes an action type sequence corresponding to at least one time window in the timing of the audio sample;

[0051] A model training unit is configured to train an expression driven model based on the first training data, wherein the training includes: taking the audio sample and the expression control parameters corresponding to the audio sample as inputs of the expression driven model, obtaining an expression parameter sequence output by the expression driven model, and updating the model parameters of the expression driven model using a loss function corresponding to a first training objective, wherein the first training objective includes minimizing the difference between the expression parameter sequence output by the expression driven model and the corresponding true value of the expression parameter.

[0052] In a fourth aspect, an expression driving device is provided, the device comprising:

[0053] A data acquisition unit, configured to acquire audio data and expression control data, wherein the expression control data at least includes timing control data, and the timing control data includes an action type sequence corresponding to at least one time window in the timing of the audio data;

[0054] An expression driving unit is configured to input the audio data and the expression control data into an expression driving model, and obtain an expression parameter sequence predicted by the expression driving model using the audio data and the expression control data, wherein the expression parameter sequence corresponds to the time sequence of the audio data;

[0055] The expression control unit is configured to control the facial expression of the virtual character based on the expression parameter sequence.

[0056] In a fifth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods in the first aspect are implemented.

[0057] In a sixth aspect, an electronic device is provided, including:

[0058] one or more processors; and

[0059] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the first aspects above.

[0060] In a seventh aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the steps of any one of the methods described in the first aspect.

[0061] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0062] 1) This application introduces timing control data so that the trained expression-driven model can accurately capture the action type sequence in a certain time window within the audio timing, thereby synchronously generating an expression parameter sequence aligned with the audio. Compared with the traditional expression-driven model that can only be controlled based on global labels, it improves the refined control effect of expression. This fine-grained timing control significantly improves the model's ability to capture expression changes, thereby generating richer and more expressive facial expressions.

[0063] 2) The present application can also predict the expression parameter sequence based on the global control data, and realize the global control of the expression by using emotion category labels, speaker style labels, and emotion intensity, thereby realizing the control of the emotion category, speaker style, emotion intensity, etc. of the expression, thereby improving the overall expressiveness and richness of the expression.

[0064] 3) This application achieves a refined control effect of expression through a specific network architecture (audio coding network, global coding network, timing coding network and diffusion network), by introducing a timing coding network, and introducing the feature representation sequence corresponding to the timing control data into the denoising processing and expression prediction of the diffusion network.

[0065] 4) This application introduces a two-stage training method. In the first training stage, the global control ability of the expression-driven model is trained. In the second training stage, the expression-driven model's perception ability of timing control data is trained, thereby improving the expression-driven model's ability to fine-tune expression control and achieving fine-grained facial movement control.

[0066] 5) This application realizes the encoding of emotion categories, speaker styles and emotion intensity through the label encoding subnetwork and intensity embedding module. In particular, the intensity embedding module integrates the control information of emotion intensity into the feature representation of global control data, further enhancing the overall expressiveness and richness of expression.

[0067] 6) The design of the adaptation module of the present application enables the expression driving model to better integrate the timing control data, thereby achieving a refined control effect of the expression and improving the accuracy and richness of the expression generation.

[0068] 7) The present application can obtain timing control data based on action description samples, so that the timing control data that can be used by the expression-driven model can be easily obtained through natural language description, thereby simplifying user operations.

[0069] 8) This application introduces a contrastive learning mechanism to pre-train the action detection module, so that the action detection module can improve the generalization ability of the action detection module while ensuring the prediction effect of the timing control data.

[0070] 9) This application introduces a masking mechanism in the inference stage of the expression-driven model. The adaptation module masks the intervals without timing control parameters, thereby preventing the diffusion problem of refined control in the expression prediction process and further improving the refined control effect of expressions.

[0071] Of course, any product implementing the present application does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0073] Figure 1 A flowchart of a method for training an expression-driven model provided in an embodiment of the present application;

[0074] Figure 2 A schematic structural diagram of the expression drive model provided in the embodiment of the present application;

[0075] Figure 3 A schematic diagram of a global coding network provided in an embodiment of the present application;

[0076] Figure 4 A schematic diagram of a diffusion model provided in an embodiment of the present application;

[0077] Figure 5 A schematic diagram of the principle of a diffusion model introducing an adaptation module provided in an embodiment of the present application;

[0078] Figure 6 A schematic diagram of a training action detection module provided in an embodiment of the present application;

[0079] Figure 7 A flowchart of the expression driving method provided in an embodiment of the present application;

[0080] Figure 8 A schematic diagram of the expression drive model provided in the embodiment of the present application;

[0081] Fig. 9 An example diagram of the expression driving method provided in the embodiment of the present application;

[0082] Fig.10 A schematic block diagram of a training device for an expression-driven model provided in an embodiment of the present application;

[0083] Fig.11 A schematic block diagram of an expression driving device provided in an embodiment of the present application;

[0084] Fig.12 A schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0085] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0086] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0087] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0088] There are already some technologies that perform cross-attention processing on the emotion type label and the feature representation corresponding to the audio in the expression-driven model, so as to predict the expression parameter sequence corresponding to the audio timing. However, this method cannot achieve fine control of the expression, resulting in poor richness and generation of the final generated expression.

[0089] In view of this, the present application provides a new idea and a new expression driving model. Figure 1 The flowchart of the training method of the expression-driven model provided in the embodiment of the present application, the execution subject of the method can be a model training device, which can be set in any computer device with data storage and processing capabilities, can be set in the server side, and can also be set in a computer terminal with strong data processing capabilities. The method may include the following steps:

[0090] Step 101: Acquire first training data including a plurality of first training samples, wherein the first training samples at least include: an audio sample, expression control data corresponding to the audio sample, and a true value of an expression parameter, wherein the expression control data at least includes timing control data, wherein the timing control data includes an action type sequence corresponding to at least one time window in the timing of the audio sample;

[0091] Step 102: training an expression driven model based on the first training data, wherein the training includes: taking an audio sample and expression control parameters corresponding to the audio sample as input of the expression driven model, obtaining an expression parameter sequence output by the expression driven model, and updating the model parameters of the expression driven model using a loss function corresponding to a first training objective, wherein the first training objective includes minimizing the difference between the expression parameter sequence output by the expression driven model and the corresponding true value of the expression parameter.

[0092] It can be seen that by introducing timing control data, the trained expression-driven model can accurately capture the action type sequence in a certain time window within the audio timing, thereby synchronously generating an expression parameter sequence aligned with the audio. Compared with the traditional expression-driven model that can only be controlled based on global labels, the refined control effect of expression is improved. This fine-grained timing control significantly improves the model's ability to capture expression changes, thereby generating richer and more expressive facial expressions.

[0093] The following is a detailed description of each step in the above process and the effects that can be further produced in conjunction with the embodiments. It should be noted that the limitations such as "first" and "second" involved in the present disclosure do not have limitations in terms of size, order, and quantity, and are only used to distinguish them in name. For example, "first training data" and "second training data" are only used to distinguish two types of training data in name. For another example, "first training target" and "second training target" are only used to distinguish two training targets in name.

[0094] First, the above step 101, namely “obtaining first training data including a plurality of first training samples” is described in detail in conjunction with the embodiment.

[0095] The first training sample includes at least: an audio sample, expression control data corresponding to the audio sample, and a true value of an expression parameter, wherein the expression control data includes at least timing control data, and the timing control data includes an action type sequence corresponding to at least one time window within the timing of the audio sample.

[0096] In the embodiment of the present application, it is first necessary to collect or generate a series of audio samples, which can be recordings of human voices or other types of audio inputs, which will serve as basic data for model training. The audio samples can cover a variety of speech content, including different speakers, intonations, speaking speeds, and emotional expressions, to ensure that the model can be generalized to various practical application scenarios.

[0097] For each audio sample, corresponding expression control data is required, and these expression control data are used to guide the expression drive model on how to generate corresponding facial expressions according to the audio content. In an embodiment of the present application, the expression control data at least includes timing control data, i.e., an action type sequence within the timing of the audio sample, and the action type sequence refers to the specific action type that the face should perform within a specific time window, such as smiling, laughing, frowning, grinning, etc. The action type sequence can correspond to at least one time window within the timing of the audio sample. For example, if the audio sample is 20 seconds in length, the timing control data can be: the 3rd to 8th seconds correspond to laughing, which can be presented as an action type sequence.

[0098] In addition to audio samples and expression control data, each training sample also needs to be provided with true values ​​of expression parameters, which can be manually annotated or obtained through high-precision facial capture equipment. The true values ​​of expression parameters are used to characterize the precise parameters of facial expressions at each time point of the audio sample, including but not limited to lip shape, eye openness, eyebrow position, etc.

[0099] The audio sample, expression control data and expression parameter true value are combined to form a first training sample. A plurality of first training samples constitute the first training data.

[0100] Furthermore, the expression control data may include global control data in addition to the timing control data. The global control data may include one or any combination of emotion category labels, speaker style labels, and emotion intensity.

[0101] The following is a detailed description of the components of global control data:

[0102] 1) Emotion category label: It is a metadata used to describe the main emotion expressed in the audio sample. Common emotion categories include happiness, sadness, anger, surprise, fear, and neutral. These labels indicate the basic emotional tone that the model needs to consider when generating facial expressions, so that the generated expressions can reflect the emotional content in the audio.

[0103] 2) Speaker style labels: used to describe the unique expression or style of a specific speaker, such as formal, casual, passionate or calm. These labels help the expression-driven model capture and imitate the personalized facial movements and expression habits of a specific individual, making the generated animation more personalized.

[0104] 3) Emotion intensity label: describes the intensity of the emotion expressed in the audio sample. It can be a quantitative indicator, such as a scale from 1 to 5 or 1 to 10, or a qualitative description, such as weak, medium, strong. With the information of the emotion intensity label, the expression-driven model can learn how to adjust the intensity of the expression to match the slight or drastic changes in the emotion in the audio.

[0105] As one of the feasible ways, the first training sample can be inferred from a real video, such as a video of a real person speaking. Audio can be extracted from the video as an audio sample, and a sequence of expression parameters can be extracted from the real person's expression in the video as the true value of the expression parameter. The timing control data and global control data of the real person in the video can be analyzed to construct the first training sample.

[0106] The above step 102, namely "training the expression driven model based on the first training data", is described in detail below in conjunction with an embodiment.

[0107] The training process may include: taking audio samples and expression control parameters corresponding to the audio samples as inputs of an expression-driven model, obtaining an expression parameter sequence output by the expression-driven model, and updating the model parameters of the expression-driven model using a loss function corresponding to a first training objective. The first training objective includes minimizing the difference between the expression parameter sequence output by the expression-driven model and the corresponding true value of the expression parameter.

[0108] As one possible implementation method, the expression driving model in the embodiment of the present application is as follows: Figure 2 As shown in , it can include an audio coding network, a global coding network, a temporal coding network and a diffusion network.

[0109] Among them, the audio coding network is used to encode the audio samples to obtain the feature representation sequence corresponding to the audio samples.

[0110] The global encoding network is used to encode the global control data to obtain the feature representation corresponding to the global control data.

[0111] The timing coding network is used to encode the timing control data to obtain a feature representation sequence corresponding to the timing control data.

[0112] The diffusion network is used to denoise the noise feature sequence using the feature representation sequence corresponding to the audio sample, the feature representation corresponding to the global control data and the feature representation corresponding to the timing control data, and use the feature sequence obtained by the denoising process to predict the expression parameter sequence.

[0113] As one of the feasible methods, the above-mentioned expression driven model can be trained in one training stage, that is, the audio sample, global control data and timing control data in the first training sample are taken as input at the same time to obtain the expression parameter sequence predicted by the expression driven model. Then, in each round of iteration, the value of the loss function is determined by using the difference between the expression parameter sequence and the corresponding true value of the expression parameter, and the model parameters of the expression driven model are updated using the value of the loss function until the preset training stop condition is reached.

[0114] The training stop condition may be, for example, that the value of the loss function is less than or equal to a preset threshold, or that a preset number of iterations is obtained.

[0115] As another achievable manner, the training process of the expression-driven model may include two training stages: a first training stage and a second training stage.

[0116] In the first training phase, the audio samples and global control data are first used as inputs of the expression-driven model, the difference between the expression parameter sequence output by the expression-driven model and the corresponding expression parameter true value is used to determine the value of the first loss function, and the model parameters of the expression-driven model are updated using the value of the first loss function. When the training stop condition is met, the second training phase begins.

[0117] In the second training stage, an adaptation module is added to at least one Transformer layer of the diffusion network, and the adaptation module is used to introduce the feature representation corresponding to the timing control data into the denoising process; the audio sample, the random global control data and the timing control data are used as the input of the expression driving model, and the difference between the expression parameter sequence output by the expression driving model for at least one time window and the corresponding expression parameter true value is used to determine the value of the second loss function, and the second loss function value is used to update only the model parameters of the adaptation module in the expression driving model. After the training stop condition is reached, the final expression driving model is obtained. The specific composition and principle of the adaptation module will be described in detail in the subsequent embodiments.

[0118] The above-mentioned first training stage uses global control data as the input of the expression driving model, which belongs to coarse-grained control capability, while the second training stage introduces timing control data as the input of the expression driving model, which belongs to fine-grained control capability. That is to say, the global control capability of the expression driving model is trained in the first training stage, and the perception ability of the expression driving model to the timing control data is trained in the second training stage, thereby improving the expression driving model's refined control ability over expressions and realizing fine-grained facial movement control.

[0119] As one possible implementation method, the global coding network in the embodiment of the present application is as follows: Figure 3 As shown in , it can be composed of two parts: the label encoding subnetwork and the intensity embedding module.

[0120] Among them, the label encoding subnetwork is used to encode the emotion category label and the speaker style label to obtain the feature representation of the emotion category label and the feature representation of the speaker style label respectively.

[0121] As one of the feasible ways, the label encoding sub-network can adopt a model such as CLIP (Contrastive Language-Image Pre-training) model.

[0122] The intensity embedding module is used to scale the intensity embedding vector according to the intensity of the emotion. In the embodiment of the present application, the intensity of the emotion can be represented by a learnable vector (referred to as the intensity embedding vector in the embodiment of the present application). The intensity embedding vector will be scaled accordingly according to the intensity of the emotion during the model training process, that is, the value of the emotion intensity is multiplied by the intensity embedding vector (i.e. Figure 3 In ), and obtain the feature representation of emotion intensity; finally, the feature representation of emotion category label, the feature representation of speaker style label and the feature representation of emotion intensity are fused (i.e. Figure 3 In ), and obtain the feature representation corresponding to the global control data, that is, Figure 3 C f At the beginning of training, the intensity embedding vector is initialized and continuously optimized during the training process.

[0123] The diffusion model is usually implemented based on the Transformer layer. In the diffusion model, each Transformer layer contains a self-attention processing layer and a cross-attention processing layer, such as Figure 4 As shown in . For the latent vector sequence output by the previous layer of Transformer (usually the vector obtained by each layer of Transformer processing is called the latent vector sequence), after self-attention processing, for example, FiLM (Feature-wise Linear Modulation) self-attention can be used, that is, the FiLM layer and the self-attention layer are combined to enhance the understanding and processing capabilities of the latent vector sequence. Then, the latent vector sequence is processed by self-attention with the feature representation corresponding to the global control data. For example, AdaLN (Adaptive Layer Normalization) self-attention can be used, that is, the AdaLN layer is used to adaptively normalize the Query, Key and Value vectors so that they have similar distributions, and the normalized Query and Key vectors are used to calculate the similarity score. Among them, the AdaLN layer performs normalization processing based on the feature representation of the global control data. This method can improve the performance and stability of the model when processing sequence data. The obtained latent vector sequence is further processed with the feature representation of the audio sample for cross-attention processing (for example, FiLMcross-attention can be used) to obtain the latent vector sequence output by the current layer of Transformer.

[0124] It should be noted here that in the first training process, the embodiment of the present application can freeze the model parameters of the label encoding subnetwork and the audio encoding network, and only update the model parameters of other networks (including the diffusion network and the intensity embedding module).

[0125] The adaptation module introduced in the second training stage is the adaptation network that enters each Transformer layer, such as Figure 5 As shown in , it can include a cross-attention layer (for example, FiLM cross-attention can be used) and a zero convolution layer. The cross-attention layer represents the feature corresponding to the temporal control data C a The hidden vector sequence output by the Transformer layer is cross-attention processed, and the hidden vector sequence obtained by the cross-attention processing is provided to the zero convolution layer for optimization. Among them, the zero convolution layer can be initialized as a convolution layer with all 0s, and is gradually updated during the training process of the second training stage. The zero convolution layer helps fine-grained conditions, i.e., time-series control data, to be more stably injected into the expression-driven model.

[0126] In the second training phase, only the model parameters of the adaptation module can be updated, and other model parameters can be frozen. In other words, by training the adaptation module in the second training phase, the dominant control of fine-grained conditions, i.e., time-series control data, is ensured.

[0127] It should be noted that the expression drive model in the embodiment of the present application can be implemented based on the Transformer architecture, and the model is trained and reasoned through the diffusion process to generate accurate expression coefficients. The expression drive model design in the embodiment of the present application can include multiple Transformer layers, for example, 8 Transformer layers, each of which includes multiple attention mechanism layers. These attention mechanism layers are responsible for processing different types of control data, including audio data, coarse-grained global control data (covering speaker style, emotion category, emotion intensity) and fine-grained timing control data, so as to generate fine expression coefficients. Through this design, the expression drive model in the embodiment of the present application can comprehensively consider the complex interaction of audio and multiple control data to achieve accurate control of expression generation. This multi-level, multi-data processing method enables the model to capture and reproduce the subtle changes of human facial expressions more carefully when generating expressions, improving the realism and expressiveness of expression animation.

[0128] It should be noted that a label exchange mechanism is essentially adopted in the above-mentioned second training stage, that is, random global control data is used in the training process, for example, the emotion category label is a random label, and the speaker style label is a random label. In this way, at the beginning of the training of the second training stage, the model will generate an expression parameter sequence based on the label. Since it is random, there is a large difference between the true value of the expression parameter. When designing the second loss function in the second training stage, only the difference between the expression parameter sequence and the corresponding true value of the expression parameter is considered in the timing window corresponding to the timing control parameter, so that the adaptation module learns in the second training stage to use the fine-grained control condition, that is, the timing control parameter, as the dominant control prediction result.

[0129] As one of the achievable ways, the above-mentioned timing control data can be directly in the form of an action type sequence.

[0130] As another achievable method, the above-mentioned timing control data can also be predicted from the action description text. For example, firstly, the action description text is obtained; then, the action detection module is used to predict the action type sequence corresponding to at least one time window based on the action description text.

[0131] That is, the action detection module can convert the description text in natural language form input by the user into an action type sequence. The action detection module can be pre-trained in the following way:

[0132] Acquire second training data including a plurality of sample pairs, wherein the sample pairs include two action description texts, and the second training data also includes action type sequence samples corresponding to each action description sample;

[0133] The method uses the second training data to train an action detection module, including: inputting an action description text into the action detection module, the action detection module extracting feature representations from the action description text, and predicting an action type sequence using the extracted feature representations; and updating the parameters of the action detection module using the loss function corresponding to the second training target, wherein the second training target includes: minimizing the distance between the feature representations of two action description texts in a positive sample pair, maximizing the distance between the feature representations of two action description texts in a negative sample pair, and minimizing the difference between the predicted action type sequence and the corresponding action type sequence sample.

[0134] Among them, when training the action detection module, two loss functions can be used:

[0135] On the one hand, the loss function is to ensure the accuracy of prediction, such as Figure 6 L shown in BCE , the purpose of this loss function is to minimize the difference between the predicted action type sequence and the corresponding action type sequence samples.

[0136] The loss function on the other hand is based on the contrastive learning mechanism to improve the generalization ability of the action detection module. The sample pairs include positive sample pairs and negative sample pairs. The two action description texts contained in the positive sample pairs correspond to the same action type sequence sample, and the two action description texts contained in the negative sample pairs correspond to different action type sequence samples. The corresponding loss function is as follows: Figure 6 L InjfoNCE The purpose of this loss function is to minimize the distance between the feature representations of the two action description texts in the positive sample pair and maximize the distance between the feature representations of the two action description texts in the negative sample pair.

[0137] As one possible implementation, the action detection module may include a text encoding subnetwork, an adaptation subnetwork, and a decoding subnetwork. The text encoding subnetwork is used to encode the input action description text, and the encoding result is input into the adaptation subnetwork for optimization to obtain the feature representation of the action description text. The decoding subnetwork uses the feature representation of the action description text to predict the action type sequence.

[0138] Based on the expression driving model obtained through the above training, the embodiment of the present application also provides an expression driving method, such as Figure 7 As shown, the following steps are included:

[0139] Step 701: Acquire audio data and expression control data, where the expression control data at least includes timing control data, and the timing control data includes an action type sequence corresponding to at least one time window in the timing of the audio data;

[0140] Step 702: input the audio data and the expression control data into the expression driving model, and obtain the expression parameter sequence predicted by the expression driving model using the audio data and the expression control data, wherein the expression parameter sequence corresponds to the time sequence of the audio data;

[0141] Step 703: Control the facial expression of the virtual character based on the expression parameter sequence.

[0142] Among them, in step 701, it is first necessary to collect audio data. This data can be voice recordings or other types of audio inputs. They will serve as the basic input of the expression-driven model, generate an expression parameter sequence for the audio data, and then generate an image sequence containing a virtual character based on the expression parameter sequence.

[0143] In addition, it is also necessary to obtain the expression control data corresponding to the audio data. The expression control data should at least include the timing control data, that is, the action type sequence corresponding to each time window in the time series of the audio data. These action type sequences are used to characterize the specific actions that the face should perform within a specific time window, such as smiling, frowning, etc.

[0144] Furthermore, the above expression control data also includes global control data, and the global control data includes one or any combination of emotion category labels, speaker style labels and emotion intensity.

[0145] As one of the implementable ways, a user interface may be provided to the user, and the user may input audio data and expression control data through specific components on the user interface.

[0146] As one of the achievable ways, the above-mentioned timing control data can be directly in the form of an action type sequence.

[0147] As another achievable method, the above-mentioned timing control data can also be predicted from the action description text. For example: obtain the action description text; use the action detection module to predict the action type sequence corresponding to at least one time window based on the action description text. The structure and principle of the action detection module can be found in Figure 6 and the related records in the previous embodiments.

[0148] For example, if the user inputs "from the 3rd second to the 7th second, control the facial expression to laugh", the action detection module can convert the natural language description text into an action type sequence. The action type sequence is a time series feature, and the action type corresponding to the part from the 3rd second to the 7th second is "laugh".

[0149] In step 702, the audio data and expression control data acquired in step 701 are input into the expression drive model. Figure 8 As shown in , the expression-driven model can include an audio encoding network, a global encoding network, a temporal encoding network, and a diffusion network.

[0150] Among them, the audio coding network is used to encode the audio data to obtain a feature representation sequence corresponding to the audio data.

[0151] The global encoding network is used to encode the global control data to obtain the feature representation corresponding to the global control data.

[0152] The timing coding network is used to encode the timing control data to obtain a feature representation sequence corresponding to the timing control data.

[0153] The diffusion network is used to denoise the noise feature sequence using the feature representation sequence corresponding to the audio data, the feature representation corresponding to the global control data and the feature representation corresponding to the timing control data, and use the feature sequence obtained by the denoising process to predict the expression parameter sequence.

[0154] The specific principles of the expression driving model can be found in the relevant records in the previous embodiments, which will not be elaborated here.

[0155] The structure of the global encoding network can be as follows Figure 3 As shown in , the specific principles can be found in the relevant records in the previous embodiments and will not be repeated here.

[0156] The expression driven model includes multiple Transformer layers, for example, 8 Transformer layers, each of which includes multiple attention mechanism layers. An adaptation module is also attached to at least one Transformer layer in the diffusion network, and the adaptation module is used to introduce the feature representation corresponding to the timing control data into the denoising process.

[0157] The adaptation module includes a cross-attention layer and a zero-convolution layer. The cross-attention layer performs cross-attention processing on the feature representation corresponding to the timing control data and the latent vector sequence output by the Transformer layer. The latent vector sequence obtained by the cross-attention processing is provided to the zero-convolution layer for optimization as the final latent vector sequence output by the Transformer layer, and then output to the next Transformer layer. For details, please refer to Figure 5 And the related records in the previous embodiments.

[0158] Furthermore, in order to improve the accuracy and dominance of facial expression control, the embodiment of the present application also introduces a mask-based conditional generation method based on the existing expression-driven model, that is, the adaptation module masks the timing outside the above-mentioned at least one time window in the input latent vector sequence, and then performs processing based on the feature representation corresponding to the timing control data based on the latent vector sequence after mask processing. In other words, the part of the latent vector sequence that does not belong to the current time window is masked to suppress the influence of the characteristics of these time periods on the model prediction. When performing mask processing, it is equivalent to using a mask matrix of the same size as the latent vector sequence, and the elements in the mask matrix can be 0 or 1 to control whether the denoising processing performed by the adaptation module using the feature representation of the timing control data acts on the corresponding latent vector.

[0159] In step 703, the expression parameter sequence is used to drive the facial expressions of the virtual character, so that when the corresponding image sequence is generated for the audio data, the facial expressions of the virtual character in each frame in the image sequence can accurately reflect the emotions and tone changes in the audio data, and comply with the global and refined control of the expression control parameters.

[0160] As one of the feasible ways, the expression parameter sequence can be used as one of the parameters of two-dimensional or three-dimensional image modeling to guide the generation of facial expressions in the process of two-dimensional or three-dimensional image modeling.

[0161] It should be noted that the virtual characters involved in the embodiments of the present application refer to human images or digital characters created using computer graphics, artificial intelligence and other technologies.

[0162] Through these three steps, the embodiment of the present application provides a complete solution from audio data to facial expressions of virtual characters in image sequences. This method can not only control expressions globally, but also achieve refined control of expressions, thereby improving the realism and emotional expression of virtual characters in image sequences, and can play an important role in multiple fields such as filmmaking, virtual reality, games, and social media.

[0163] Fig. 9 An example diagram of using the expression driving method provided by the embodiment of the present application is shown in FIG. Fig. 9 As shown in , the user can input audio data and expression control data. The global control data in the expression control data acts on the entire time series corresponding to the audio data, that is, the corresponding time range. For example, the user inputs the emotion type label "angry", the speaker style "senior actor" and the emotion intensity "2", as shown in Fig. 9 The timing control data can be used only for a part of the time interval (i.e., time window) in the timing corresponding to the audio data. For example, the user inputs the action type "laughing", and the corresponding time window is as follows: Fig. 9 The resulting image sequence not only reflects the emotional intensity of anger, senior actor, and 2 globally, but also reflects the expression corresponding to the action type "laughing" in the character's expression within the user-specified time window, thus achieving more refined expression control.

[0164] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0165] According to an embodiment of another aspect, a training device for an expression-driven model is provided. Fig.10 A schematic block diagram of the training device according to one embodiment is shown, and the device is arranged in any computer device with data storage and processing capabilities. Fig.10 As shown, the device 1000 includes: a sample acquisition unit 1001 and a model training unit 1002. The main functions of each component unit are as follows:

[0166] The sample acquisition unit 1001 is configured to acquire first training data including a plurality of first training samples, wherein the first training samples at least include: an audio sample, expression control data corresponding to the audio sample, and a true value of an expression parameter, wherein the expression control data at least includes timing control data, and the timing control data includes an action type sequence corresponding to at least one time window in the timing of the audio sample;

[0167] The model training unit 1002 is configured to train the expression driven model based on the first training data, wherein the training includes: taking the audio sample and the expression control parameters corresponding to the audio sample as input of the expression driven model, obtaining the expression parameter sequence output by the expression driven model, and updating the model parameters of the expression driven model using the loss function value corresponding to the first training objective, wherein the first training objective includes minimizing the difference between the expression parameter sequence output by the expression driven model and the corresponding true value of the expression parameter.

[0168] As one achievable manner, the expression control data further includes global control data, and the global control data includes one or any combination of an emotion category label, a speaker style label, and an emotion intensity.

[0169] As one of the achievable ways, the expression driven model in the model training unit 1002 includes: an audio coding network, a global coding network, a temporal coding network and a diffusion network;

[0170] The audio coding network is used to encode the audio sample to obtain a feature representation sequence corresponding to the audio sample;

[0171] The global encoding network is used to encode the global control data to obtain a feature representation corresponding to the global control data;

[0172] The timing coding network is used to encode the timing control data to obtain a feature representation sequence corresponding to the timing control data;

[0173] The diffusion network is used to denoise the noise feature sequence using the feature representation sequence corresponding to the audio sample, the feature representation corresponding to the global control data and the feature representation corresponding to the timing control data, and predict the expression parameter sequence using the feature sequence obtained by the denoising process.

[0174] As one of the implementable ways, the training in the model training unit 1002 includes a first training stage and a second training stage;

[0175] In the first training stage, the audio sample and the global control data are used as inputs of the expression drive model, a first loss function value is determined by using the difference between the expression parameter sequence output by the expression drive model and the corresponding expression parameter true value, and the model parameters of the expression drive model are updated by using the first loss function value;

[0176] In the second training stage, an adaptation module is added to at least one Transformer layer of the diffusion network, and the adaptation module is used to introduce the feature representation corresponding to the timing control data into the denoising process; the audio samples, random global control data and the timing control data are used as inputs of the expression driving model, and the value of the second loss function is determined by the difference between the expression parameter sequence output by the expression driving model for the at least one time window and the corresponding expression parameter true value, and the second loss function value is used to update only the model parameters of the adaptation module in the expression driving model.

[0177] As one of the achievable ways, the global encoding network in the model training unit 1002 includes: a label encoding subnetwork and an intensity embedding module;

[0178] The label encoding subnetwork is used to encode the emotion category label and the speaker style label to obtain a feature representation of the emotion category label and a feature representation of the speaker style label respectively;

[0179] The intensity embedding module is used to scale the intensity embedding vector according to the emotion intensity to obtain a feature representation of the emotion intensity, and the intensity embedding vector is continuously optimized during the training process;

[0180] The feature representation of the emotion category label, the feature representation of the speaker style label and the feature representation of the emotion intensity are fused to obtain the feature representation corresponding to the global control data.

[0181] As one of the feasible ways, the adaptation module in the model training unit 1002 includes a cross-attention layer and a zero convolution layer. The cross-attention layer performs cross-attention processing on the feature representation corresponding to the timing control data and the latent vector sequence output by the Transformer layer. The latent vector sequence obtained by the cross-attention processing is provided to the zero convolution layer for optimization. The zero convolution layer outputs the optimized latent vector sequence to the next Transformer layer.

[0182] As one of the achievable methods, the timing control data is obtained by:

[0183] Get the action description text;

[0184] An action detection module is used to predict an action type sequence corresponding to at least one time window based on the action description text.

[0185] As one of the achievable methods, the action detection module is pre-trained in the following manner:

[0186] Acquire second training data including a plurality of sample pairs, wherein the sample pairs include two action description texts, and the second training data also includes action type sequence samples corresponding to each action description sample;

[0187] The action detection module is trained using the second training data, including: inputting the action description text into the action detection module, the action detection module extracting feature representations from the action description text, and predicting the action type sequence using the extracted feature representations; and updating the parameters of the action detection module using the loss function value corresponding to the second training objective, wherein the second training objective includes: minimizing the distance between the feature representations of two action description texts in a positive sample pair, maximizing the distance between the feature representations of two action description texts in a negative sample pair, and minimizing the difference between the predicted action type sequence and the corresponding action type sequence sample.

[0188] According to an embodiment of another aspect, an expression driving device is provided. Fig.11 A schematic block diagram of the expression driving device according to one embodiment is shown, and the device is arranged in any computer device with data storage and processing capabilities. Fig.11 As shown, the device 1100 includes: a data acquisition unit 1101, an expression driving unit 1102 and an expression control unit 1103. The main functions of each component unit are as follows:

[0189] The data acquisition unit 1101 is configured to acquire audio data and expression control data, wherein the expression control data at least includes timing control data, and the timing control data includes an action type sequence corresponding to at least one time window in the timing of the audio data;

[0190] The expression driving unit 1102 is configured to input the audio data and the expression control data into an expression driving model, and obtain an expression parameter sequence predicted by the expression driving model using the audio data and the expression control data, wherein the expression parameter sequence corresponds to the time sequence of the audio data;

[0191] The expression control unit 1103 is configured to control the facial expression of the virtual character based on the expression parameter sequence.

[0192] As one achievable manner, the expression control data further includes global control data, and the global control data includes one or any combination of an emotion category label, a speaker style label, and an emotion intensity.

[0193] As one of the achievable methods, the expression driven model includes: an audio coding network, a global coding network, a temporal coding network and a diffusion network;

[0194] The audio coding network is used to encode the audio data to obtain a feature representation sequence corresponding to the audio data;

[0195] The global encoding network is used to encode the global control data to obtain a feature representation corresponding to the global control data;

[0196] The timing coding network is used to encode the timing control data to obtain a feature representation sequence corresponding to the timing control data;

[0197] The diffusion network is used to denoise the noise feature sequence using the feature representation sequence corresponding to the audio data, the feature representation corresponding to the global control data and the feature representation corresponding to the timing control data, and predict the expression parameter sequence using the feature sequence obtained by the denoising process.

[0198] As one of the possible implementations, the global encoding network includes: a label encoding subnetwork and an intensity embedding module;

[0199] The label encoding subnetwork is used to encode the emotion category label and the speaker style label to obtain a feature representation of the emotion category label and a feature representation of the speaker style label respectively;

[0200] The intensity embedding module is used to scale the intensity embedding vector according to the emotion intensity to obtain a feature representation of the emotion intensity, and the intensity embedding vector is continuously optimized during the training process;

[0201] The feature representation of the emotion category label, the feature representation of the speaker style label and the feature representation of the emotion intensity are fused to obtain the feature representation corresponding to the global control data.

[0202] As one of the implementable ways, an adaptation module is attached to at least one Transformer layer of the diffusion network, and the adaptation module is used to introduce the feature representation corresponding to the timing control data into the denoising process.

[0203] As one of the feasible ways, the adaptation module includes a cross-attention layer and a zero convolution layer. The cross-attention layer performs cross-attention processing on the feature representation corresponding to the timing control data and the latent vector sequence output by the Transformer layer. The latent vector sequence obtained by the cross-attention processing is provided to the zero convolution layer for optimization. The zero convolution layer outputs the optimized latent vector sequence to the next Transformer layer.

[0204] As one possible implementation method, the device further includes a mask processing unit configured to:

[0205] The adaptation module performs mask processing on the timing part outside the at least one time window in the input latent vector sequence, and processes the input latent vector sequence according to the feature representation corresponding to the timing control data.

[0206] As one achievable method, obtaining the timing control data includes:

[0207] Get the action description text;

[0208] An action detection module is used to predict an action type sequence corresponding to at least one time window based on the action description text.

[0209] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The system and device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0210] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0211] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods in the aforementioned method embodiments are implemented.

[0212] And an electronic device, comprising:

[0213] one or more processors; and

[0214] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the aforementioned method embodiments.

[0215] The present application also provides a computer program product, including a computer program, which implements the steps of any one of the methods in the aforementioned method embodiments when executed by a processor.

[0216] in, Fig.12The architecture of the electronic device is shown as an example, which may include a processor 1210, a video display adapter 1211, a disk drive 1212, an input / output interface 1213, a network interface 1214, and a memory 1220. The processor 1210, the video display adapter 1211, the disk drive 1212, the input / output interface 1213, the network interface 1214, and the memory 1220 may be communicatively connected via a communication bus 1230.

[0217] Among them, the processor 1210 can be implemented by a general-purpose CPU, a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., to execute relevant programs to implement the technical solutions provided in this application.

[0218] The memory 1220 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1220 can store an operating system 1221 for controlling the operation of the electronic device 1200, and a basic input and output system (BIOS) 1222 for controlling the low-level operation of the electronic device 1200. In addition, a web browser 1223, a data storage management system 1224, and a playback processing device 1225 for media files, etc. can also be stored. The playback processing device 1225 for the above-mentioned media files can be an application program that specifically implements the operations of the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided by the present application is implemented by software or firmware, the relevant program code is stored in the memory 1220 and is called and executed by the processor 1210.

[0219] The input / output interface 1213 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0220] The network interface 1214 is used to connect to a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0221] The bus 1230 comprises a pathway for transmitting information between the various components of the device (eg, the processor 1210 , the video display adapter 1211 , the disk drive 1212 , the input / output interface 1213 , the network interface 1214 , and the memory 1220 ).

[0222] It should be noted that, although the above device only shows a processor 1210, a video display adapter 1211, a disk drive 1212, an input / output interface 1213, a network interface 1214, a memory 1220, a bus 1230, etc., in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include components necessary for implementing the solution of the present application, and does not necessarily include all the components shown in the figure.

[0223] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a computer program product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0224] The technical solution provided by the present application is described in detail above. The principle and implementation method of the present application are described in detail using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A training method for an expression-driven model, characterized in that: The method comprises: Acquire first training data including a plurality of first training samples, wherein the first training samples at least include: an audio sample, expression control data corresponding to the audio sample, and a true value of an expression parameter, wherein the expression control data at least includes timing control data, and the timing control data includes an action type sequence corresponding to at least one time window in the timing of the audio sample; The expression driven model is trained based on the first training data, wherein the training includes: taking the audio sample and the expression control parameters corresponding to the audio sample as inputs of the expression driven model, obtaining the expression parameter sequence output by the expression driven model, and updating the model parameters of the expression driven model using the loss function corresponding to the first training objective, wherein the first training objective includes: minimizing the difference between the expression parameter sequence output by the expression driven model and the corresponding true value of the expression parameter.

2. The method according to claim 1, characterized in that The expression control data also includes global control data, and the global control data includes one or any combination of an emotion category label, a speaker style label, and an emotion intensity.

3. The method according to claim 2, characterized in that The expression driven model includes: an audio coding network, a global coding network, a temporal coding network and a diffusion network; The audio coding network is used to encode the audio sample to obtain a feature representation sequence corresponding to the audio sample; The global encoding network is used to encode the global control data to obtain a feature representation corresponding to the global control data; The timing coding network is used to encode the timing control data to obtain a feature representation sequence corresponding to the timing control data; The diffusion network is used to denoise the noise feature sequence using the feature representation sequence corresponding to the audio sample, the feature representation corresponding to the global control data and the feature representation corresponding to the timing control data, and predict the expression parameter sequence using the feature sequence obtained by the denoising process.

4. The method according to claim 3, characterized in that The training includes a first training phase and a second training phase; In the first training stage, the audio sample and the global control data are used as inputs of the expression drive model, a first loss function value is determined by using the difference between the expression parameter sequence output by the expression drive model and the corresponding expression parameter true value, and the model parameters of the expression drive model are updated by using the first loss function value; In the second training stage, an adaptation module is added to at least one Transformer layer of the diffusion network, and the adaptation module is used to introduce the feature representation corresponding to the timing control data into the denoising process; the audio samples, random global control data and the timing control data are used as inputs of the expression driving model, and the value of the second loss function is determined by the difference between the expression parameter sequence output by the expression driving model for the at least one time window and the corresponding expression parameter true value, and the second loss function value is used to update only the model parameters of the adaptation module in the expression driving model.

5. The method according to claim 3, characterized in that: The global encoding network includes: a label encoding subnetwork and an intensity embedding module; The label encoding subnetwork is used to encode the emotion category label and the speaker style label to obtain a feature representation of the emotion category label and a feature representation of the speaker style label respectively; The intensity embedding module is used to scale the intensity embedding vector according to the emotion intensity to obtain a feature representation of the emotion intensity, and the intensity embedding vector is continuously optimized during the training process; The feature representation of the emotion category label, the feature representation of the speaker style label and the feature representation of the emotion intensity are fused to obtain the feature representation corresponding to the global control data.

6. An expression driving method, characterized in that: The method comprises: Acquire audio data and expression control data, wherein the expression control data at least includes timing control data, and the timing control data includes an action type sequence corresponding to at least one time window in the timing of the audio data; Inputting the audio data and the expression control data into an expression driving model, obtaining an expression parameter sequence predicted by the expression driving model using the audio data and the expression control data, wherein the expression parameter sequence corresponds to the time sequence of the audio data; Based on the expression parameter sequence, the facial expression of the virtual character is controlled.

7. The method according to claim 6, characterized in that The expression control data also includes global control data, and the global control data includes one or any combination of an emotion category label, a speaker style label, and an emotion intensity.

8. The method according to claim 7, characterized in that The expression driven model includes: an audio coding network, a global coding network, a temporal coding network and a diffusion network; The audio coding network is used to encode the audio data to obtain a feature representation sequence corresponding to the audio data; The global encoding network is used to encode the global control data to obtain a feature representation corresponding to the global control data; The timing coding network is used to encode the timing control data to obtain a feature representation sequence corresponding to the timing control data; The diffusion network is used to denoise the noise feature sequence using the feature representation sequence corresponding to the audio data, the feature representation corresponding to the global control data and the feature representation corresponding to the timing control data, and predict the expression parameter sequence using the feature sequence obtained by the denoising process.

9. The method according to claim 8, characterized in that The global encoding network includes: a label encoding subnetwork and an intensity embedding module; The label encoding subnetwork is used to encode the emotion category label and the speaker style label to obtain a feature representation of the emotion category label and a feature representation of the speaker style label respectively; The intensity embedding module is used to scale the intensity embedding vector according to the emotion intensity to obtain a feature representation of the emotion intensity, and the intensity embedding vector is continuously optimized during the training process; The feature representation of the emotion category label, the feature representation of the speaker style label and the feature representation of the emotion intensity are fused to obtain the feature representation corresponding to the global control data.

10. The method according to claim 8, characterized in that An adaptation module is attached to at least one Transformer layer of the diffusion network, and the adaptation module includes a cross-attention layer and a zero convolution layer. The cross-attention layer performs cross-attention processing on the feature representation corresponding to the timing control data and the latent vector sequence output by the Transformer layer, and provides the latent vector sequence obtained by the cross-attention processing to the zero convolution layer for optimization. The zero convolution layer outputs the optimized latent vector sequence to the next Transformer layer.

11. The method according to claim 10, characterized in that The method further comprises: The adaptation module performs mask processing on the timing part outside the at least one time window in the input latent vector sequence, and then processes the input latent vector sequence according to the feature representation corresponding to the timing control data.

12. A training device for an expression-driven model, characterized in that: The device comprises: A sample acquisition unit is configured to acquire first training data including a plurality of first training samples, wherein the first training samples at least include: an audio sample, expression control data corresponding to the audio sample, and a true value of an expression parameter, wherein the expression control data at least includes timing control data, and the timing control data includes an action type sequence corresponding to at least one time window in the timing of the audio sample; A model training unit is configured to train an expression driven model based on the first training data, wherein the training includes: taking the audio sample and the expression control parameters corresponding to the audio sample as inputs of the expression driven model, obtaining an expression parameter sequence output by the expression driven model, and updating the model parameters of the expression driven model using a loss function corresponding to a first training objective, wherein the first training objective includes minimizing the difference between the expression parameter sequence output by the expression driven model and the corresponding true value of the expression parameter.

13. An expression driving device, characterized in that: The device comprises: A data acquisition unit, configured to acquire audio data and expression control data, wherein the expression control data at least includes timing control data, and the timing control data includes an action type sequence corresponding to at least one time window in the timing of the audio data; An expression driving unit is configured to input the audio data and the expression control data into an expression driving model, and obtain an expression parameter sequence predicted by the expression driving model using the audio data and the expression control data, wherein the expression parameter sequence corresponds to the time sequence of the audio data; The expression control unit is configured to control the facial expression of the virtual character based on the expression parameter sequence.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

15. An electronic device, characterized in that: include: one or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions, wherein when the program instructions are read and executed by the one or more processors, the steps of the method according to any one of claims 1 to 11 are executed.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images