Mouth shape animation generation method and device, mouth shape animation model training method and device, equipment and medium
Through knowledge distillation training model, lip animations are directly generated from speech features, solving the problem of high generation cost of virtual image lip animations, and achieving more efficient generation efficiency and better versatility.
Patent Information
- Application Number
- CN202311767379.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-20
- Publication Date
- 2025-01-10
AI Technical Summary
In the prior art, the generation cost of virtual image lip animation is high and the model processing speed is slow, and a cost-reducing solution is needed to improve the generation efficiency.
Using a model based on knowledge distillation training, the process is simplified, the dependence on intermediate features is reduced, and a simpler model is used to improve efficiency.
It effectively reduces the cost of lip animation generation, improves the generation efficiency, and makes the model easier to deploy in various scenarios, with better general effects.
Smart Images
Figure CN119296515A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present invention relate to the field of Internet technology, and in particular to a lip animation generation, model training method, device, equipment and medium. Background Art
[0002] In virtual application scenarios, during the interaction process, the virtual image needs to drive the virtual image's mouth shape to correspond to the played voice data.
[0003] In the prior art, when the avatar is lip-driven, the received voice data is usually analyzed to obtain the phonemes contained therein and the duration corresponding to each phoneme, and the lip shape corresponding to each phoneme is queried in the lip shape configuration table using factors. In addition, when a machine learning model is used to generate lip animation, the model is usually complex and has many parameters, resulting in slow model processing speed and high cost of generating voice-driven lip animation. Therefore, a solution is needed to reduce the cost of generating voice-driven lip animation. Summary of the invention
[0004] The embodiment of the present invention provides a lip animation generation and model training method, device, equipment and medium, which are used to realize a solution for reducing the cost of speech-driven lip animation generation.
[0005] In a first aspect, an embodiment of the present invention provides a method for generating a lip animation, the method comprising:
[0006] Get voice data;
[0007] Extracting a first speech feature of the speech data using a first model;
[0008] Based on the extracted first speech feature, lip shape data aligned with it is determined so as to generate the speech-driven lip shape animation based on the lip shape data; wherein the first model is a model obtained by performing knowledge distillation training on the second model and has the ability to generate lip shape data using speech features.
[0009] Optionally, extracting the first voice feature of the voice data by using the first model includes:
[0010] Extracting a second speech feature using the feature extractor in the first model;
[0011] After mapping the second speech feature to a high-dimensional feature space, the first speech feature containing context information is output using a first self-attention mechanism model.
[0012] Optionally, determining the lip shape data aligned with the first speech feature based on the extracted first speech feature includes:
[0013] The lip shape data aligned with the first speech feature is predicted using a multi-layer first regression network model and the first speech feature containing context information.
[0014] Optionally, determining the lip shape data aligned with the first speech feature based on the extracted first speech feature includes:
[0015] If the speech data is associated with emotion information, the lip shape data corresponding to the emotion information is determined based on the extracted first speech feature containing context information and emotion features.
[0016] Optionally, the first model includes a first self-attention mechanism model and a first regression network model;
[0017] The training method of the first model includes:
[0018] Using a first loss function corresponding to a second self-attention mechanism in the second model to perform knowledge distillation training on the self-attention mechanism to be trained in the first model, to obtain the first self-attention mechanism model; and
[0019] The second loss function corresponding to the second regression network model in the second model is used to perform knowledge distillation training on the regression network model to be trained in the first model to obtain the first regression network model.
[0020] Optionally, the method further includes: determining a window range of the sliding window;
[0021] Selecting historical speech data samples contained in the training samples using the sliding window;
[0022] The first model is optimized based on the selected training samples.
[0023] In a second aspect, an embodiment of the present invention provides a method for generating a lip animation, the method comprising:
[0024] Acquire a first speech data sample for training a first model;
[0025] The model to be trained is trained by performing knowledge distillation using the first speech data sample and the second model obtained by pre-training, so as to obtain a first model capable of converting speech data into first lip shape data; wherein the first model after knowledge distillation training learns from the second model the ability to generate lip shape data using speech features.
[0026] Optionally, the method of performing knowledge distillation training on the to-be-trained model using the first speech data sample and the second model obtained by pre-training to obtain a first model capable of converting speech data into first lip shape data includes:
[0027] Using a first loss function corresponding to a second self-attention mechanism in the second model to perform knowledge distillation training on the self-attention mechanism to be trained in the first model, to obtain the first self-attention mechanism model; and
[0028] Using a second loss function corresponding to the second regression network model in the second model, performing knowledge distillation training on the regression network model to be trained in the first model to obtain the first regression network model;
[0029] The training obtains a model including the first self-attention mechanism model, the first regression network model and the first model; wherein the first model has fewer network layers and parameters than the second model.
[0030] Optionally, after obtaining the first model for converting the voice data into lip shape data, the method further includes:
[0031] Acquire a second voice data sample carrying the first voice data sample and corresponding emotion information;
[0032] The first model is optimized and trained using the second voice data sample so that the third voice features in the optimized third model are aligned with the first voice features, and the second lip shape data is aligned with the first lip shape data, so that the third model has the ability to generate lip shape data using voice data carrying emotional information.
[0033] In a third aspect, an embodiment of the present invention provides a lip animation generation device, the device comprising:
[0034] An acquisition module, used for acquiring voice data;
[0035] An extraction module, configured to extract a first speech feature of the speech data using a first model; wherein the first speech feature is a speech feature containing context information;
[0036] A determination module is used to determine the lip shape data aligned with the first speech feature extracted, so as to generate the speech-driven lip animation based on the lip shape data; wherein the first model is a model obtained by performing knowledge distillation training on the second model and has the ability to generate lip shape data using speech features.
[0037] In a fourth aspect, an embodiment of the present invention further provides an electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor to implement the method described in the first aspect or the method described in the second aspect.
[0038] In a fourth aspect, an embodiment of the present invention further provides a computer-readable medium on which is stored at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by a processor to implement the method described in the first aspect or the method described in the second aspect.
[0039] In an embodiment of the present invention, speech data is acquired; a first speech feature of the speech data is extracted using a first model; based on the extracted first speech feature, lip shape data aligned therewith is determined so as to generate the speech-driven lip shape animation based on the lip shape data; wherein the first model is a model having the ability to generate lip shape data using speech features based on knowledge distillation training of the second model. Through the above scheme, the received speech data is directly converted into lip shape data using a simpler first model without the need to convert it into an intermediate feature carrier (e.g., phoneme), thereby effectively improving the efficiency of lip shape animation generation. In addition, the first model used here is trained using knowledge distillation and has the ability to accurately convert speech features into lip shape data. At the same time, the model is smaller and more convenient to deploy in various scenarios, and has a better general effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0041] Figure 1 A flow chart of a method for generating a speech-driven lip animation according to an embodiment of the present application;
[0042] Figure 2 A schematic diagram of a first model processing flow for illustrating an embodiment of the present application;
[0043] Figure 3 A schematic diagram of a lip animation processing flow containing emotional information provided in an embodiment of the present application;
[0044] Figure 4 A flowchart of a model training method provided in an embodiment of the present application;
[0045] Figure 5 A schematic diagram of the first model training structure provided in an embodiment of the present application;
[0046] Figure 6 A schematic diagram of the structure of a lip animation generation device provided in an embodiment of the present application;
[0047] Figure 7 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;
[0048] Figure 8 For Figure 6 , Figure 7 A schematic diagram of the structure of an electronic device corresponding to the apparatus provided in the embodiment. DETAILED DESCRIPTION
[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0050] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two.
[0051] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.
[0052] In addition, the step sequence in the following method embodiments is only an example and not a strict limitation.
[0053] In a virtual interactive scene (e.g., a game production scene), different game virtual objects (e.g., virtual characters, personified virtual animals, virtual plants, cartoon characters, or other fictional objects) are matched with appropriate lip animations to represent the voice interaction capabilities of the virtual object. The obtained lip animations are usually displayed to the audience in conjunction with the voice playback process. In the prior art, some solutions require that after obtaining the voice data, it is first necessary to parse it to obtain the phonemes and the duration corresponding to each phoneme, and then use the pre-prepared phoneme lip configuration table to find the lip data corresponding to each phoneme, and then drive the lip shape of the virtual object to perform the corresponding animation action according to the lip data found within the duration. Another solution is to use the audio information obtained by parsing the voice data to calculate the lip change rate, and then combine the global and local preset expression weights to generate the lip animation that drives the virtual object. Another solution is to construct a universal lip feature point migration function, migrate the universal lip feature points to realize the migration of the lip feature point sequence of any audio to the lip feature points of the specified virtual object, and then generate the lip animation that drives the virtual object. Of course, there is also a machine learning model that converts the parsed phonemes into corresponding lip animations. The above-mentioned methods are relatively complicated to generate lip animation, and require intermediate features such as phonemes to obtain lip animation. Therefore, a method for simplifying the lip animation generation method is needed, which can directly and quickly generate lip animation based on voice data.
[0054] To facilitate understanding, specific embodiments will be used as examples below.
[0055] like Figure 1 The flowchart of the method for generating speech-driven lip animation provided in the embodiment of the present application is shown in FIG. Figure 1 As you can see, the specific steps include the following:
[0056] Step 101: Acquire voice data.
[0057] Step 102: Extracting a first speech feature of the speech data using a first model.
[0058] Step 103: Determine the lip shape data aligned with the first extracted speech feature, so as to generate the speech-driven lip shape animation based on the lip shape data; wherein the first model is a model having the ability to generate lip shape data using speech features based on knowledge distillation training of the second model.
[0059] like Figure 2The first model processing flow diagram is an example of the present application embodiment. The first model here is suitable for feature extraction of various voice data, and the extracted first voice feature is a universal voice feature. Therefore, there is no restriction on the type of voice data. For example, the voice data can be: real voice, virtual voice, male voice, female voice, Chinese voice, English voice and other voice data from different sources. The lip shape data mentioned here can be, for example, lip shape data driven by 55-dimensional Blendshapes.
[0060] In the present application, after obtaining the first voice feature, the first model is further used to determine the lip shape data aligned with each first voice feature in the voice data. In this process, there is no need to convert the voice data into intermediate features such as phonemes, but to directly align the voice features with the lip shape data. Once the voice features are extracted, the lip shape data aligned with them is known.
[0061] For example, for the acquired speech data, MFCC (Mel frequency cepstral coefficient) can be used as a common speech feature extraction method. MFCC can capture the spectral features of speech and encode them. By applying MFCC to speech data, we can obtain a series of feature vectors, each of which represents the spectral features of a short segment of speech. Each feature vector usually contains some information related to the speech spectrum, such as the intensity of the sound, frequency distribution, etc. In the training stage, the paired lip animation data is used to train the alignment model. For each speech feature vector, it is input into the alignment model to be trained, and the alignment model learns to map the speech features to the corresponding 55-dimensional Blendshapes weights. After the alignment model is trained, the first model is obtained. The trained first model can be a neural network model, which is trained by learning the relationship between the speech features in the training data and the corresponding Blendshapes weights. In the application stage, when new speech data is input, the speech data is firstly feature extracted to obtain a series of first speech feature vectors. Then, these first speech feature vectors are input into the trained first model to obtain the corresponding 55-dimensional Blendshapes weights. These weights describe the degree of change in facial expressions. Finally, by interpolating or fusing the weights with the initial facial shape of the virtual object, lip animation data that matches the speech can be generated. These lip animation data can be applied to the character model to drive the character's facial expression to be consistent with the input speech. For given speech data, a series of speech feature vectors can be obtained through feature extraction methods (such as MFCC). Then, these feature vectors are converted into corresponding 55-dimensional Blendshapes weights through the first model, which describes the degree of change in facial expression. Finally, these weights are applied to the facial model that drives the character to generate lip animation data that matches the speech. It can effectively improve the efficiency of lip animation generation.
[0062] In addition, the first model mentioned here is a model obtained by knowledge distillation of the second model. Therefore, the first model is simpler than the second model and is easier to deploy in various application scenarios, such as online environments and mobile devices with limited computing power. Although the first model is simpler than the second model, the first model has learned from the second model the ability to directly convert speech features into lip data, and no longer needs to convert speech data into intermediate feature data (such as phonemes). Therefore, it can significantly improve the generation efficiency of lip animation.
[0063] In one or more embodiments of the present application, the extracting the first speech feature of the speech data using the first model includes: extracting the second speech feature using the feature extractor in the first model; mapping the second speech feature to a high-dimensional feature space, and then outputting the first speech feature containing context information using the first self-attention mechanism model.
[0064] In practical applications, the convolutional neural network sub-model (CNN_feature_extractor) in the first model is first used to preliminarily extract the second speech features. Specifically, the convolutional neural network sub-model accepts the original speech data as input, and extracts the second speech features representing the low-level features of the speech through a series of convolution and pooling operations. These low-level features can capture the local structure and spectral information of the speech data (that is, the speech signal). Then the preliminarily extracted second speech features are mapped to a high-dimensional feature space through the mapping function (Feature_projector). Through this mapping process, the speech features can be converted into a more expressive representation to better capture the relationship between the semantics of the speech and the speech features. Further, the encoder model (Transformer_encoder) based on the self-attention mechanism is used to extract general speech features with global context information. This step accepts the first speech features after feature projection as input, and integrates and encodes the information through multi-layer self-attention and feedforward neural networks. In this way, long-distance dependencies and global context information in the speech sequence can be captured, and richer speech representations can be generated.
[0065] It should be noted that the speech feature extraction model uses a large amount of various types of speech data for pre-training, so that the network can extract better general speech features. Specifically, during the pre-training stage, these models will learn general speech feature representations, so that they can perform well in different speech data type scenarios. The pre-trained model for speech feature extraction can be used as the basic model for subsequent speech-driven lip animation tasks, providing rich and accurate speech information for accurate alignment of lip data and generation of lip animation.
[0066] Through the above solution, the first model can be applied to extract features from various types of speech data, and the extracted first speech features representing common speech features can be better used for subsequent alignment of lip shape data, which helps to improve alignment efficiency and alignment accuracy.
[0067] In one or more embodiments of the present application, determining the lip shape data aligned with the extracted first speech feature based on the first speech feature includes: using a multi-layer first regression network model and the first speech feature containing context information to predict the lip shape data aligned with the first speech feature.
[0068] It should be noted that speech features with global context information can capture long-distance dependencies and global semantic information in speech sequences. This context information is very important for aligning lip data, because changes in facial expressions are usually affected by the entire speech sequence. By using Transformer Encoder to extract speech features with global context information, the relationship between the semantics of speech and speech features can be better captured, thereby achieving a more accurate alignment effect. However, phonemes cannot reflect context information.
[0069] In practical applications, based on the extraction of the first speech feature containing context information, a first regression network model with a multi-layer regression network is designed to regress the speech feature vector to a 55-dimensional Blendshapes coefficient vector. Specifically,
[0070] In the part of aligning speech features with lip data, it is necessary to design a first regression network model with a multi-layer regression network to regress the speech feature vector with global context information to a 55-dimensional Blendshapes coefficient vector. This regression network can be a deep neural network consisting of multiple hidden layers, each of which contains multiple neurons. The input of the network is the speech feature vector with global context information, and the output is the predicted Blendshapes coefficient vector.
[0071] It should be noted that the first regression network model uses paired speech data and corresponding Blendshapes coefficient vectors to train the regression network during training. For each speech feature vector, it is input into the regression network, and the predicted Blendshapes coefficient vector is obtained through the forward propagation process of the network. Then, a loss function (such as mean square error) is used to compare the difference between the predicted coefficient vector and the true coefficient vector, and the weights and biases of the network are updated through the back propagation algorithm, so that the network can gradually optimize and reduce the prediction error. Through the trained first regression network model, new speech features with global context information can be input into the network to obtain the corresponding predicted Blendshapes coefficient vector. These coefficient vectors describe the degree of change of facial expressions. By applying these coefficients to the facial model of the driving character, lip animation data matching the speech can be generated to achieve alignment of speech and facial expressions. In the above process, context information globally models the entire speech sequence, and plays an important role in capturing the relationship between semantics and speech features. The introduction of this global context information can improve the accuracy and coherence of alignment.
[0072] In one or more embodiments of the present application, determining the lip shape data aligned with the first extracted speech feature includes: if the speech data is associated with emotional information, then determining the lip shape data corresponding to the emotional information based on the extracted first speech feature containing contextual information and emotional features.
[0073] like Figure 3 A schematic diagram of the lip animation processing flow containing emotional information provided in the embodiment of the present application. In some application scenarios, the emotions of the virtual object need to be reflected. Therefore, when generating the lip animation, emotional information should be added, that is, different emotional voices are needed to drive different emotional lip animations to ensure that the emotional voice and lip animation are consistent. Figure 3 As can be seen in the figure, the emotional speech driven lip animation model is obtained by further optimizing the first model mentioned above. In the model training stage, the emotional information is input into the model to be trained, so as to obtain the lip data that can reflect the emotional information. Specifically, after the first model (ATF model) is obtained through training, the frozen Figure 2The parameters of the CNN_feature_extractor and Feature_projector parts in the first model shown are used, and the emotion information Emotion_Embeddings is added for training. The input part uses speech data with the same corpus but different emotions, and applies the corresponding learnable emotion vector to the standard speech features in the feature space, and minimizes the MSE loss to achieve the alignment of the standard speech and emotional speech in the feature space. In addition, the learnable emotion vector is also applied to the standard speech features finally output by the Transformer Encoder, and the MSE loss is minimized to achieve the alignment of the final regression results. When using the model trained in the above manner, input the standard speech and choose to superimpose the emotion vector of the corresponding emotion to drive the lip animation of the corresponding emotion.
[0074] For example, when training the model, use speech data with the same corpus but different emotions as input. Extract standard speech features from the input speech through the CNN_feature_extractor and Feature_projector parts. Introduce Emotion_Embeddings as emotion vectors and apply them to standard speech features in the feature space to achieve alignment of standard speech and emotional speech in the feature space. Minimize the mean square error (MSE) loss function to make the aligned standard speech features close to emotional speech features. When outputting lip animation, input speech data with emotion information. During the processing, superimpose the emotion vector on the first speech feature. Use Transformer Encoder to process the first speech feature superimposed with the emotion vector and obtain the final regression result. Minimize the MSE loss function to make the final regression result close to the emotional speech feature. Through the above scheme, the trained model can align speech data, speech features corresponding to different emotions with lip data, and drive lip animation according to the selected emotion during inference. In this way, the effect of expressing different emotions in lip animation can be achieved.
[0075] In one or more embodiments of the present application, the first model includes a first self-attention mechanism model and a first regression network model. The training method of the first model includes: using the first loss function corresponding to the second self-attention mechanism in the second model to perform knowledge distillation training on the self-attention mechanism to be trained in the first model to obtain the first self-attention mechanism model; and using the second loss function corresponding to the second regression network model in the second model to perform knowledge distillation training on the regression network model to be trained in the first model to obtain the first regression network model.
[0076] The second model mentioned here is the teacher model, and the corresponding first model is the student model. The performance and generalization ability of the student model are improved by transferring the knowledge of the teacher model to the student model. The teacher model is trained using a large amount of training data. The teacher model can be a deep model with a high number of parameters and complexity. Through training, the teacher model learns the complex mapping relationship between the input speech and the corresponding lip animation. In this method, the teacher model is usually a larger and more complex model, while the student model is a compressed lightweight model.
[0077] In actual applications, the teacher model (that is, the second model) includes Feature_projector, Regression_Network, CNN_feature_extractor and Transformer Encoder. Among them, Feature_projector and Regression_Network share the same structure with the teacher model. This part has a simple structure and does not consume much computing power during the application process. Therefore, it does not need to be learned by the student model, but can be used directly.
[0078] However, for the second regression network model (CNN_feature_extractor) and the second self-attention mechanism model (Transformer Encoder) in the feature extraction of the teacher model, the student model needs to learn the feature extraction ability of the teacher model after knowledge distillation training. The specific knowledge distillation training process is as follows:
[0079] The student model is trained using the same training dataset. The network structure of the student model is usually simpler than that of the teacher model, such as reducing the number of layers or the number of parameters. During the training process, in addition to using real labeled data for training, the prediction results of the teacher model are also used as an additional objective function. During the training of the student model, the knowledge distillation loss function is introduced to measure the difference between the prediction results of the student model and the prediction results of the teacher model. This can be achieved by calculating the distance between the two or other similarity measures. The introduction of the knowledge distillation loss function helps the student model better learn the knowledge of the teacher model.
[0080] By introducing the knowledge distillation loss function, the student model can learn the teacher model's ability in feature extraction, that is, learn how to extract useful features from speech. At the same time, since the student model's network structure is compressed, the student model is lighter than the teacher model, with a smaller number of network layers and size. The knowledge distillation training process involves using the teacher model's prediction results as an additional objective function to help the student model learn the teacher model's knowledge. By introducing the knowledge distillation loss function, the student model can maintain similar performance to the teacher model with a smaller number of network layers and size, and benefit from the teacher model's knowledge in feature extraction.
[0081] During the training process, the window range of the sliding window is determined; the historical speech data samples contained in the training samples are selected using the sliding window; and the first model is optimized based on the selected training samples.
[0082] Specifically, during the training process, the local receptive field restriction is imposed on the attention matrix of the model, that is, the restriction is achieved by adding a sliding window. The range of the restriction can be reflected by the window range, which is the data within a certain window size range in the history and future, such as the data in the last week.
[0083] When processing streaming data, the data arrives sequentially, and the model needs to make predictions at each time step. In order to enable the model to make predictions for the input of the current time step while taking into account past contextual information, an attention mechanism can be used. The attention mechanism allows the model to give different weights to different positions in the input sequence when generating predictions, depending on their importance. In this way, the model can focus on information relevant to the current time step when generating predictions. However, in the case of streaming data, it is hoped that the model can look back at features within the historical window to better understand the context and the evolution of the sequence. To achieve this, a local receptive field restriction is imposed. The local receptive field restriction means that the model can only focus on a small part of the input sequence, i.e., a local region, when calculating attention. This local region can be a historical window around the current time step. By limiting the scope of attention, the model can only rely on information in the local region when generating predictions, thereby better capturing the contextual relationships in the streaming data. This restriction helps improve the streaming reasoning effect of the model, because the model does not rely too much on the information of the entire input sequence, but pays more attention to the local features related to the current time step. In this way, the model can better adapt to the characteristics of streaming data and make real-time predictions and decisions.
[0084] Based on the same idea, the present application embodiment also proposes a model training method. Figure 4 A flow chart of the model training method provided in the embodiment of the present application. Figure 4As can be seen in , the method specifically includes the following steps:
[0085] Step 401: Obtain a first speech data sample for training a first model.
[0086] Step 402: Perform knowledge distillation training on the model to be trained using the first speech data sample and the second model obtained through pre-training to obtain a first model capable of converting speech data into first lip shape data; wherein the first model after knowledge distillation training learns from the second model the ability to generate lip shape data using speech features.
[0087] As described in step 402, the first speech data sample and the second model obtained by pre-training are used to perform knowledge distillation training on the to-be-trained model, so as to obtain a first model capable of converting speech data into first lip shape data, including:
[0088] Step 4021: Perform knowledge distillation training on the self-attention mechanism to be trained in the first model using the first loss function corresponding to the second self-attention mechanism in the second model to obtain the first self-attention mechanism model. And,
[0089] Step 4022: Perform knowledge distillation training on the regression network model to be trained in the first model using the second loss function corresponding to the second regression network model in the second model to obtain the first regression network model.
[0090] Step 4023: training to obtain a model including the first self-attention mechanism model, the first regression network model, and the first model; wherein the first model has fewer network layers and parameters than the second model.
[0091] As mentioned above, the teacher model (that is, the second model) is obtained using a large amount of training data. The teacher model can be a deep model with a high number of parameters and complexity. Through training, the teacher model learns the complex mapping relationship between the input voice data and the corresponding lip animation. Use the trained teacher model to infer the input voice data and obtain the prediction results of the teacher model. These prediction results can be corresponding weights or other information that characterizes the lip animation. The reasoning ability of the teacher model is accurate enough, but the teacher model is usually a larger and more complex model, which requires high computing power and high deployment scenarios during data processing. When the hardware capabilities are limited, the data processing efficiency of the teacher model will be relatively low. Therefore, there is a need for a compressed, lightweight student model (that is, the first model) that can be easily deployed and has faster data processing efficiency.
[0092] like Figure 5A schematic diagram of the first model training structure provided in the embodiment of the present application. The first model compression scheme is based on the idea of knowledge distillation, which aims to achieve model compression and acceleration by transferring the knowledge of the teacher model to the student model. The specific scheme is as follows:
[0093] The Feature_projector and Regression_Network parts of the student model share the same structure as the teacher model, to ensure that the student model can learn similar representation capabilities as the teacher model in the task of feature mapping and alignment. The CNN_feature_extractor and Transformer Encoder parts have compressed the number of network layers and sizes. This means that the student model will use a simpler network structure than the teacher model during feature extraction to achieve model compression and acceleration.
[0094] In the knowledge distillation process during training, the self-supervised network part uses KL loss to realize the knowledge transfer from the teacher model to the student model. KL loss is a loss function that measures the difference between two probability distributions. In knowledge distillation, the student model is trained to approximate the behavior of the teacher model by calculating the difference between the output probability distributions of the teacher model and the student model in the task of feature mapping and alignment. The regression network part uses MSE loss to realize the knowledge transfer on the final prediction result. MSE loss is used to measure the mean square error between the prediction result and the true label. In knowledge distillation, the student model is guided to learn the knowledge of the teacher model on the task by calculating the mean square error between the prediction results of the teacher model and the student model on the regression task.
[0095] Through this model compression scheme based on knowledge distillation, the student model can obtain knowledge from the teacher model and compress it in size and number of layers to reduce the model size and increase the inference speed. This method shows potential in model compression and acceleration, and can use smaller models to complete complex tasks on computing resources or mobile devices.
[0096] In one or more embodiments of the present application, after obtaining the first model capable of converting voice data into lip shape data, it also includes: obtaining a second voice data sample carrying the first voice data sample and corresponding emotional information; optimizing and training the first model using the second voice data sample, so that the third voice feature in the third model obtained after optimization is aligned with the first voice feature, and the second lip shape data is aligned with the first lip shape data, so that the third model has the ability to generate lip shape data using voice data carrying emotional information.
[0097] from Figure 3As can be seen in the figure, the emotional speech driven lip animation model (that is, the third model) is obtained by further optimizing the first model mentioned above. In the model training stage, the second voice data sample carrying the first voice data sample and the corresponding emotional information is input into the model to be trained, so as to obtain lip data that can reflect the emotional information. Specifically, after the first model (ATF model) is obtained through training, the freeze Figure 2 The parameters of the CNN_feature_extractor and Feature_projector parts in the first model shown are used, and the emotion information Emotion_Embeddings is added for training. The input part uses speech data with the same corpus but different emotions, and applies the corresponding learnable emotion vector to the standard speech features in the feature space, and minimizes the MSE loss to achieve the alignment of the first speech feature and the third speech feature in the feature space. In addition, the first speech feature finally output by the Transformer Encoder is also applied with a learnable emotion vector, and the MSE loss is minimized to achieve the alignment of the final regression results, that is, to achieve the alignment of the third speech feature with the first speech feature, and the alignment of the second lip shape data with the first lip shape data. When using the model trained in the above manner, input speech data and choose to superimpose the emotion information of the corresponding emotion to drive the lip shape animation of the corresponding emotion.
[0098] Based on the same idea, the present application also provides a lip animation generation device. Figure 6 A schematic diagram of the structure of a lip animation generation device provided in an embodiment of the present application. Figure 6 As can be seen, the device includes:
[0099] The acquisition module 61 is used to acquire voice data.
[0100] The extraction module 62 is used to extract the first speech feature of the speech data using the first model; wherein the first speech feature is a speech feature containing context information.
[0101] The determination module 63 is used to determine the lip shape data aligned with the first speech feature extracted, so as to generate the speech-driven lip shape animation based on the lip shape data; wherein the first model is a model obtained by performing knowledge distillation training on the second model and has the ability to generate lip shape data using speech features.
[0102] An extraction module 62, configured to extract a second speech feature using a feature extractor in the first model;
[0103] After mapping the second speech feature to a high-dimensional feature space, the first speech feature containing context information is output using a first self-attention mechanism model.
[0104] The extraction module 62 is used to predict the lip shape data aligned with the first speech feature by using a multi-layer first regression network model and the first speech feature containing context information.
[0105] The extraction module 62 is configured to determine, if the speech data is associated with emotion information, lip shape data corresponding to the emotion information based on the extracted first speech feature containing context information and emotion features.
[0106] The first model includes a first self-attention mechanism model and a first regression network model.
[0107] Optionally, a training module 64 is further included, which is used to train the first model. The training method includes:
[0108] Using a first loss function corresponding to a second self-attention mechanism in the second model to perform knowledge distillation training on the self-attention mechanism to be trained in the first model, to obtain the first self-attention mechanism model; and
[0109] The second loss function corresponding to the second regression network model in the second model is used to perform knowledge distillation training on the regression network model to be trained in the first model to obtain the first regression network model.
[0110] A training module 64, used to determine the window range of the sliding window;
[0111] Selecting historical speech data samples contained in the training samples using the sliding window;
[0112] The first model is optimized based on the selected training samples.
[0113] Based on the same idea, the present application embodiment also provides a model training device. Figure 7 A schematic diagram of the structure of a model training device provided in an embodiment of the present application. Figure 7 As can be seen, the device includes:
[0114] The acquisition module 71 is used to acquire a first speech data sample for training a first model.
[0115] The training module 72 is used to perform knowledge distillation training on the model to be trained using the first speech data sample and the second model obtained by pre-training, so as to obtain a first model capable of converting speech data into first lip shape data; wherein, the first model after knowledge distillation training learns from the second model the ability to generate lip shape data using speech features.
[0116] A training module 72 is used to perform knowledge distillation training on the self-attention mechanism to be trained in the first model using a first loss function corresponding to the second self-attention mechanism in the second model to obtain the first self-attention mechanism model; and
[0117] Using a second loss function corresponding to the second regression network model in the second model, performing knowledge distillation training on the regression network model to be trained in the first model to obtain the first regression network model;
[0118] The training obtains the first model including the first self-attention mechanism model, the first regression network model and a small number of network layers and parameters; wherein the first model has fewer network layers and parameters than the second model.
[0119] A training module 72, configured to obtain a second voice data sample carrying the first voice data sample and corresponding emotion information;
[0120] The first model is optimized and trained using the second voice data sample so that the third voice features in the optimized third model are aligned with the first voice features, and the second lip shape data is aligned with the first lip shape data, so that the third model has the ability to generate lip shape data using voice data carrying emotional information.
[0121] Based on the above embodiment, voice data is obtained; a first voice feature of the voice data is extracted using a first model; based on the extracted first voice feature, lip shape data aligned therewith is determined so as to generate the voice-driven lip shape animation based on the lip shape data; wherein the first model is a model having the ability to generate lip shape data using voice features based on knowledge distillation training of the second model. Through the above scheme, the received voice data is directly converted into lip shape data using a simpler first model without the need to convert it into an intermediate feature carrier (e.g., phoneme), thereby effectively improving the efficiency of lip shape animation generation. In addition, the first model used here is trained using knowledge distillation and has the ability to accurately convert voice features into lip shape data. At the same time, the model is smaller and more convenient to deploy in various scenarios, and has a better general effect.
[0122] In one possible design, the above Figure 6 The structure of the lip animation generating device shown, and Figure 7 The structure of the model training device shown in the figure can be realized as an electronic device. Figure 8As shown, the electronic device may include: a processor 81 and a memory 82. The memory 82 stores executable code, and when the executable code is executed by the processor 81, the processor 81 can at least implement the visual effect enhancement method provided in the above embodiment. The structure of the electronic device may also include a communication interface 83 for communicating with other devices or communication networks.
[0123] In addition, an embodiment of the present invention provides a non-transitory machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor in a server or a computer, the processor executes the operations provided in the above embodiments. Figure 1 The corresponding lip animation generation method and Figure 4 The corresponding model training method.
[0124] The device embodiments described above are merely illustrative, wherein the modules described as separate components may or may not be physically separated. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. Those of ordinary skill in the art may understand and implement the present invention without creative effort.
[0125] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by adding a necessary general hardware platform, and of course can also be implemented by combining hardware and software. Based on such an understanding, the above technical solution can essentially or contribute to the part in the form of a computer product, and the present invention can be implemented in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for generating lip animation, characterized in that: The method comprises: Get voice data; Extracting a first speech feature of the speech data using a first model; Based on the extracted first speech feature, lip shape data aligned with it is determined so as to generate the speech-driven lip shape animation based on the lip shape data; wherein the first model is a model obtained by performing knowledge distillation training on the second model and has the ability to generate lip shape data using speech features.
2. The method according to claim 1, characterized in that The extracting the first voice feature of the voice data by using the first model includes: Extracting a second speech feature using the feature extractor in the first model; After mapping the second speech feature to a high-dimensional feature space, the first speech feature containing context information is output using a first self-attention mechanism model.
3. The method according to claim 1, characterized in that The step of determining the lip shape data aligned with the first speech feature based on the extracted first speech feature comprises: The lip shape data aligned with the first speech feature is predicted using a multi-layer first regression network model and the first speech feature containing context information.
4. The method according to claim 1, characterized in that: The step of determining the lip shape data aligned with the first speech feature based on the extracted first speech feature comprises: If the speech data is associated with emotion information, the lip shape data corresponding to the emotion information is determined based on the extracted first speech feature containing context information and emotion features.
5. The method according to claim 1, characterized in that The first model includes a first self-attention mechanism model and a first regression network model; The training method of the first model includes: Using the first loss function corresponding to the second self-attention mechanism in the second model to perform knowledge distillation training on the self-attention mechanism to be trained in the first model, to obtain the first self-attention mechanism model; and The second loss function corresponding to the second regression network model in the second model is used to perform knowledge distillation training on the regression network model to be trained in the first model to obtain the first regression network model.
6. The method according to claim 5, characterized in that Also includes: Determine the window range of the sliding window; Selecting historical speech data samples contained in the training samples using the sliding window; The first model is optimized based on the selected training samples.
7. A model training method, characterized in that: The method comprises: Acquire a first speech data sample for training a first model; The model to be trained is trained by performing knowledge distillation using the first speech data sample and the second model obtained by pre-training, so as to obtain a first model capable of converting speech data into first lip shape data; wherein the first model after knowledge distillation training learns from the second model the ability to generate lip shape data using speech features.
8. The method according to claim 7, characterized in that The method of performing knowledge distillation training on the to-be-trained model by using the first speech data sample and the second model obtained by pre-training to obtain a first model capable of converting speech data into first lip shape data comprises: Using the first loss function corresponding to the second self-attention mechanism in the second model to perform knowledge distillation training on the self-attention mechanism to be trained in the first model, to obtain the first self-attention mechanism model; and Using a second loss function corresponding to the second regression network model in the second model, performing knowledge distillation training on the regression network model to be trained in the first model to obtain the first regression network model; The training obtains a model including the first self-attention mechanism model, the first regression network model and the first model; wherein the first model has fewer network layers and parameters than the second model.
9. The method according to claim 7, characterized in that: After obtaining the first model for converting the voice data into the lip data, the method further includes: Acquire a second voice data sample carrying the first voice data sample and corresponding emotion information; The first model is optimized and trained using the second voice data sample so that the third voice features in the optimized third model are aligned with the first voice features, and the second lip shape data is aligned with the first lip shape data, so that the third model has the ability to generate lip shape data using voice data carrying emotional information.
10. A speech-driven lip animation generation device, characterized in that: The device comprises: An acquisition module, used for acquiring voice data; An extraction module, configured to extract a first speech feature of the speech data using a first model; wherein the first speech feature is a speech feature containing context information; A determination module is used to determine the lip shape data aligned with the first speech feature extracted, so as to generate the speech-driven lip animation based on the lip shape data; wherein the first model is a model obtained by performing knowledge distillation training on the second model and has the ability to generate lip shape data using speech features.
11. An electronic device comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 9.
12. A computer-readable medium having stored thereon at least one instruction, at least one program, code set or instruction set, wherein the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the method according to any one of claims 1 to 9.