Mouth shape animation generation method and device, mouth shape animation model training method and device, equipment and medium

Abstract

The invention discloses a mouth shape animation generation method and device, a model training method and device, equipment and a medium. The method comprises the following steps: acquiring text information, a virtual object identifier and an emotion identifier; converting the text information into a phoneme sequence, converting the virtual object identifier into a virtual object code, and converting the emotion identifier into an emotion code; inputting the phoneme sequence, the virtual object code and the emotion code into a coding model for feature combination to obtain a comprehensive feature for representing common features of the voice and the mouth shape; and decoding the comprehensive feature to obtain a voice signal corresponding to the text information and a matched mouth shape signal, so as to generate a mouth shape animation by using the voice signal and the mouth shape signal. According to the scheme, the input text information, the virtual object identifier and the emotion identifier are processed through the coding model to obtain the comprehensive characteristics representing the common characteristics of the voice and the mouth shape, and then the voice signal and the mouth shape signal corresponding to the same text information are obtained to generate the mouth shape animation, so that the working efficiency of generating the mouth shape animation through the text information is improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of Internet technology, and in particular to lip animation generation, model training methods, devices, equipment and media. Background Art

[0002] In a virtual interactive scene, users can watch the various actions of virtual targets in the scene, and some users can also interact with virtual objects to gain a greater sense of participation.

[0003] Sometimes, what can be provided is text information, and then the virtual object is required to generate a lip animation based on the received text information and play the voice matching the lip shape at the same time. In the prior art, if you want to output a voice signal and automatically play the corresponding lip animation, it usually needs to be divided into two processing stages, first converting the text information into voice information, and then converting the voice information into a lip animation. These two stages require their own algorithms and models, and the processing flow is complicated and inefficient. Therefore, a solution is needed to reduce the generation cost of generating lip animation based on text information. Summary of the invention

[0004] The embodiments of the present invention provide a lip animation generation, model training method, device, equipment and medium for improving the sensory experience effect of users in virtual interaction.

[0005] In a first aspect, an embodiment of the present invention provides a method for generating a lip animation, the method comprising:

[0006] Obtaining text information, virtual object identifiers, and emotion identifiers;

[0007] Converting the text information into a phoneme sequence, converting the virtual object identifier into a virtual object code, and converting the emotion identifier into an emotion code;

[0008] Inputting the phoneme sequence, the virtual object code, and the emotion code into a coding model for feature combination to obtain a comprehensive feature for characterizing common features of speech and lip shape;

[0009] The comprehensive features are decoded to obtain a voice signal and a matching lip shape signal corresponding to the text information, so as to generate a lip shape animation using the voice signal and the lip shape signal.

[0010] Optionally, the inputting the phoneme sequence, the virtual object code, and the emotion code into a coding model for feature combination to obtain a comprehensive feature for characterizing common features of speech and lip shape includes:

[0011] Inputting the phoneme sequence, the virtual object code and the emotion code into a text feature model to obtain text features;

[0012] Inputting the text features into a mapping module for feature combination to obtain low-dimensional features;

[0013] Inputting the text features into a time prediction module for processing to obtain time prediction information;

[0014] Normalization processing is performed based on the virtual object code, the emotion code, the low-dimensional feature and the time prediction information to obtain the comprehensive feature.

[0015] Optionally, the step of inputting the text features into a mapping module for feature combination to obtain low-dimensional features includes:

[0016] The text features including audio features, virtual object features representing constraint conditions and emotional features are input into a mapping module for feature concatenation and / or feature weighting to obtain low-dimensional features for representing the correlation relationship among the audio features, virtual object features and the emotional features.

[0017] Optionally, the inputting the text features into a time prediction module for processing to obtain time prediction information includes:

[0018] The text features are input into the time prediction module to obtain prediction information for representing the phoneme duration in the low-dimensional features, so as to align the speech signal with the lip signal based on the prediction information.

[0019] Optionally, the performing normalization processing based on the virtual object code, the emotion code, the low-dimensional feature and the time prediction information to obtain the comprehensive feature includes:

[0020] Inputting the virtual object code and the emotion code into at least one flow layer in a flow module;

[0021] inputting the time prediction information into a designated flow layer;

[0022] In the flow module, the low-dimensional features, the virtual object code, and the emotion code are normalized to obtain comprehensive features that can express the emotions of the virtual object.

[0023] Optionally, decoding the comprehensive feature to obtain a speech signal corresponding to the text information and a matching lip signal includes:

[0024] inputting the comprehensive features into a first decoding model and a second decoding model respectively;

[0025] Obtaining a lip shape weight coefficient output by the first decoding model and a speech signal output by the second decoding model;

[0026] A lip shape signal matching the speech signal is obtained based on the lip shape weight coefficient.

[0027] In a second aspect, an embodiment of the present invention provides a model training method, the method comprising:

[0028] Obtaining training samples consisting of text information, virtual object identifiers, and emotion identifiers;

[0029] Inputting the training samples into the model to be trained so that the trained machine learning model has encoding and decoding capabilities;

[0030] Among them, the encoding model in the machine learning model is used to output comprehensive features for characterizing the common characteristics of speech and lip shape based on the input text information, virtual object identifier, and emotion identifier; the decoding model in the machine learning model is used to decode the speech signal and the matching lip shape signal based on the comprehensive features.

[0031] Optionally, the model to be trained includes a lip rendering model to be trained;

[0032] The training method of the model to be trained includes:

[0033] Inputting the comprehensive training features obtained during the training process into the decoding model to be trained to obtain a first loss corresponding to the lip decoding weight coefficient;

[0034] and inputting the lip decoding weight coefficient into the lip rendering model to be trained to obtain a second loss;

[0035] The model to be trained is iteratively optimized based on the first loss and the second loss.

[0036] Optionally, the model to be trained also includes a discriminator;

[0037] The training method of the model to be trained includes:

[0038] The trained speech signal and lip shape signal are input into the corresponding discriminator respectively to obtain the classification prediction loss that represents whether it is a real signal and the feature map loss extracted based on the signal;

[0039] The model to be trained is iteratively optimized using the classification prediction loss and the feature map loss.

[0040] Optionally, the model to be trained includes a time prediction module to be trained;

[0041] The training method of the time prediction module to be trained includes:

[0042] Obtaining training comprehensive features and training phoneme sequences;

[0043] Establishing a corresponding relationship between the training comprehensive feature and the training phoneme sequence;

[0044] The training comprehensive features and the training phoneme sequence having a corresponding relationship are input into the time prediction module to be trained, so that the trained time prediction module outputs prediction information representing the phoneme duration.

[0045] In a third aspect, an embodiment of the present invention provides a lip animation generation device, the device comprising:

[0046] An acquisition module, used to acquire text information, virtual object identifiers, and emotion identifiers;

[0047] A conversion module, used to convert the text information into a phoneme sequence, convert the virtual object identifier into a virtual object code, and convert the emotion identifier into an emotion code;

[0048] An encoding module, used for inputting the phoneme sequence, the virtual object code, and the emotion code into an encoding model for feature combination to obtain a comprehensive feature for characterizing common features of speech and lip shape;

[0049] A decoding module is used to decode the comprehensive features to obtain a voice signal and a matching lip shape signal corresponding to the text information, so as to generate a lip shape animation using the voice signal and the lip shape signal.

[0050] In a fourth aspect, an embodiment of the present invention further provides an electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor to implement the lip animation generation method according to the first aspect or the model training method according to the second aspect.

[0051] In a fourth aspect, an embodiment of the present invention further provides a computer-readable medium having stored thereon at least one instruction, at least one program, a code set or an instruction set, wherein the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by a processor to implement the lip animation generation method according to the first aspect or the model training method according to the second aspect.

[0052] In an embodiment of the present invention, text information, virtual object identifiers, and emotion identifiers are obtained; the text information is converted into a phoneme sequence, the virtual object identifier is converted into a virtual object code, and the emotion identifier is converted into an emotion code; the phoneme sequence, the virtual object code, and the emotion code are input into a coding model for feature combination to obtain a comprehensive feature for characterizing the common features of speech and lip shape; the comprehensive feature is decoded to obtain a speech signal and a matching lip shape signal corresponding to the text information, so as to generate a lip shape animation using the speech signal and the lip shape signal. Through the above scheme, the input text information, virtual object identifier, and emotion identifier are converted and processed by the coding model to obtain a comprehensive feature that can be used to characterize the common features of speech and lip shape, without the cumbersome process of converting text into speech and then converting to obtain lip shape data. After obtaining the comprehensive feature, the comprehensive feature is decoded to obtain a speech signal and a corresponding lip shape signal corresponding to the same text information, and then a lip shape animation containing speech is generated, thereby improving the work efficiency of directly generating lip shape animation from text information, and the obtained speech and lip shape animation have good consistency. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 A flowchart of a lip animation generation method provided in an embodiment of the present application;

[0055] Figure 2 A schematic diagram of the coding module workflow provided in the embodiment of the present application;

[0056] Figure 3 A schematic diagram of a process for adding virtual objects and emotion coding features provided in an embodiment of the present application;

[0057] Figure 4 A schematic diagram of the discriminator training process provided in an embodiment of the present application;

[0058] Figure 5 A schematic diagram of lip shape processing provided in an embodiment of the present application;

[0059] Figure 6 A flowchart of a model training method provided in an embodiment of the present application;

[0060] Figure 7 A schematic diagram of the lip animation generation process provided in an embodiment of the present application;

[0061] Figure 8 A schematic diagram of the structure of a lip animation generation device provided in an embodiment of the present application;

[0062] Fig. 9 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0063] Fig.10 For Figure 8 , Fig. 9 A schematic diagram of the structure of an electronic device corresponding to the apparatus provided in the embodiment. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two.

[0066] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0067] In addition, the step sequence in the following method embodiments is only an example and not a strict limitation.

[0068] In virtual interactive scenes, some virtual objects need to be able to understand text information, and then convert the text information into lip animation with voice, and play the voice information while displaying the lip animation through the virtual object. However, in the prior art, this conversion process is quite complicated and is mainly divided into two parts. The first part is to convert text into voice, and the second part is to convert the voice into the lip animation of the virtual object. The first part and the second part use different models respectively, and the model structure is complex. In addition, if different models and different algorithms are used to generate voice and lip animation respectively, the generation process is cumbersome, and the consistency and correlation between voice and lip animation are poor. Therefore, a solution is needed that can directly generate voice and lip animation using text information.

[0069] To facilitate understanding, specific embodiments will be used as examples below.

[0070] like Figure 1 The flowchart of the lip animation generation method provided in the embodiment of the present application is as follows. Figure 1 As you can see, the specific steps include the following:

[0071] Step 101: Obtain text information, virtual object identifier, and emotion identifier.

[0072] Step 102: converting the text information into a phoneme sequence, converting the virtual object identifier into a virtual object code, and converting the emotion identifier into an emotion code.

[0073] Step 103: Input the phoneme sequence, the virtual object code, and the emotion code into a coding model for feature combination to obtain a comprehensive feature for characterizing the common features of speech and lip shape.

[0074] Step 104: Decode the comprehensive features to obtain a voice signal and a matching lip shape signal corresponding to the text information, so as to generate a lip shape animation using the voice signal and the lip shape signal.

[0075] The text information mentioned here can be understood as various natural language texts such as Chinese and English provided by users or servers, or machine languages ​​that can be used to represent natural languages ​​(for example, computer languages ​​in binary form). In some scenarios, text information can be provided directly by users or servers; in other scenarios, such as scenarios where people interact with virtual objects, text information is generated when people respond to voice information or text information provided by people.

[0076] The virtual objects mentioned here are various virtual images such as virtual characters, virtual animals, virtual cartoons, etc., which are used to be displayed to users. The virtual objects can be displayed to users in an anthropomorphic form, can have a mouth for speaking like a human, and the mouth shape can correspond to the voice, so that users can obtain a more realistic interactive experience. The virtual object identifier can be understood as an identifier used to distinguish different virtual objects. Since virtual objects can be of various types, such as virtual objects for the elderly, virtual objects for children, virtual objects for cartoon animals, virtual objects for cartoon plants, etc.; different types of virtual objects have different timbre, tone, expression, etc. when speaking. When generating voice based on text information, it is necessary to generate the corresponding voice according to the virtual object identifier. If this virtual object identifier is not set, the default value (for example, 0 or empty) is used when it is used. When selecting a virtual object identifier, the corresponding identifier can be selected by looking up a table. The identifier can be in the form of numbers, words, letters, alphanumeric combinations, etc.

[0077] The emotional identifiers mentioned here can be understood as identifiers corresponding to various emotional states. When the same text content is converted into lip animations, there are differences in lip animations under different emotional states. The emotional states mentioned here can be, for example, calm, angry, happy, sad, angry, and so on. Each emotional state has a unique identifier for distinction. During the application process, you only need to provide the corresponding identifier to know what the emotional state of the current virtual object is. If the emotional identifier is not set, the default value (for example, 0 or empty) is used when it is used. When selecting an emotional identifier, you can select the corresponding identifier by looking up the table. The identifier can be in the form of numbers, words, letters, alphanumeric combinations, and so on.

[0078] Phoneme is the smallest unit of speech, which represents the sound in language, such as the pronunciation of phoneme. In order to facilitate the conversion of speech into phonemes, a phoneme dictionary can be created. When converting speech into phonemes, the conversion can be done by querying the dictionary.

[0079] The way to convert text information into phoneme sequences is as follows:

[0080] The input text is tokenized and each word is appropriately labeled. This allows each word to be mapped to a corresponding phoneme sequence in subsequent steps. Each word obtained after tokenizing the text information is converted into a corresponding phoneme sequence according to a phoneme dictionary. This can be achieved by looking up the phoneme representation of each word in the dictionary, or using more complex rules or machine learning models to perform the conversion. For the generated phoneme sequence, some post-processing operations can be performed, such as merging consecutive identical phonemes, processing stress and syllable boundaries, etc. It should be noted that phoneme conversion is a complex task, and text conversion systems of different language types may have different processing methods. Some advanced text conversion systems convert text information into phonemes in order to generate speech and lip animation more accurately. This process converts text into basic units of speech, allowing the system to better extract details of pronunciation and speech in order to find common features of speech and lip animation.

[0081] The virtual object coding and emotion coding mentioned here are coded data that can be processed by the coding model. This data is obtained after conversion based on the virtual object identifier and emotion identifier.

[0082] After the phoneme sequence, virtual object code, and emotion code are input into the coding model, the coding model is used for coding processing to obtain a comprehensive feature that can characterize the common features of speech and lip shape. In other words, after decoding processing using the comprehensive feature, speech and lip shape can be obtained, and the two are completely corresponding, and there is no need to filter, align, match, etc. the speech and lip shape animation. Therefore, using the above solution, the lip shape animation generation speed is faster and more efficient.

[0083] It should be noted that the lip animation mentioned here includes the visual animation picture and the voice corresponding to the lip shape in the picture. The final lip animation can be applied to various virtual object displays and interactive scenes.

[0084] In one or more embodiments of the present application, Figure 2 A schematic diagram of the coding module workflow provided in an embodiment of the present application. The process of inputting the phoneme sequence, the virtual object code, and the emotion code into the coding model for feature combination to obtain a comprehensive feature for characterizing the common features of speech and lip shape specifically includes the following steps:

[0085] Step 201: input the phoneme sequence, the virtual object code and the emotion code into a text feature model to obtain text features.

[0086] Step 202: Input the text features into a mapping module for feature combination to obtain low-dimensional features.

[0087] Step 203: Input the text features into a time prediction module for processing to obtain time prediction information.

[0088] Step 204: Perform normalization processing based on the virtual object code, the emotion code, the low-dimensional feature and the time prediction information to obtain the comprehensive feature.

[0089] In practical applications, the input phoneme sequence, virtual object encoding, and sentiment encoding are fed into a text feature model. This model is used to extract feature representations related to the text. It can be a deep learning model, such as a recurrent neural network (RNN) or a convolutional neural network (CNN), which can capture contextual information and semantic associations in the input data. The output is a text feature vector that reflects the content of the text. The mapping module receives the text features obtained in the previous step and performs feature combination on them. The purpose of this mapping module is to project the text features into a low-dimensional representation space to extract more compact and relevant information. It can use linear transformations, nonlinear activation functions, and other techniques to achieve feature combination and dimensionality reduction.

[0090] In the time prediction module, text features are taken as input and time prediction processing is performed. The task of this time prediction module is to predict the time length of speech generation and lip movement. It can use regression models or sequence models to learn the mapping relationship between text features and time length. The time prediction module is trained to predict the duration of each phoneme, or predict lip movement within a specific time period. The output is the time prediction information associated with the input text. Using the time prediction module, the output speech can be aligned with the lip matching and have better time consistency. The step of normalizing virtual object encoding, emotion encoding, low-dimensional features and time prediction information usually involves mapping the input data to a fixed range or standardization to ensure that the weights of different features are balanced and eliminate the proportional differences of different features.

[0091] A comprehensive feature representation is obtained through normalization processing based on virtual object encoding, emotion encoding, low-dimensional features and time prediction information. The comprehensive features are used to jointly characterize the characteristics of speech and lip shape. This comprehensive feature vector contains virtual object information, emotion information, low-dimensional text features and time-related prediction information to comprehensively describe the common characteristics between speech and lip shape. In order to obtain matching speech signals and lip shape signals at the same time by decoding the comprehensive features. This comprehensive feature vector can simultaneously integrate multiple input information such as text information, virtual objects and emotions to more comprehensively and accurately describe the characteristics of speech and lip shape, providing valuable input for subsequent tasks and applications.

[0092] It should be noted that the models and algorithms in the specific implementation may vary depending on the task. The above steps provide a general framework for illustrating how to integrate multiple input information into a comprehensive feature. In practice, the specific model architecture and data processing steps may need to be adjusted according to specific application scenarios and requirements. The following will illustrate the achievable solutions through specific embodiments, which does not constitute a limitation on the achievable solutions of this application.

[0093] For example, first, the phoneme sequence and virtual character encoding (equivalent to the virtual object encoding in the previous text) and emotion encoding are input into the [text encoder] (text encoding module) for text feature encoding. The obtained text features are input into the time prediction module [Duration predictor] to obtain time prediction information (i.e., the time length of speech and lip shape); and at the same time, they are input into the mapping module [projection] to generate the probability distribution of low-dimensional features through the input text feature vector. These low-dimensional feature probability distributions can be represented by a parameterized probability distribution, usually using the normal distribution (also called Gaussian distribution). That is, the text features are further combined to obtain the mean μ and variance σ of the fitted Z distribution. The character emotion encoding, Z distribution estimation and time information are input into the normalization flow module [flow] to improve the feature expression ability and integrate to obtain the final feature latent Z.

[0094] In one or more embodiments of the present application, the inputting the text features into a mapping module for feature combination to obtain low-dimensional features includes:

[0095] The text features including audio features, virtual object features representing constraint conditions and emotional features are input into a mapping module for feature concatenation and / or feature weighting to obtain low-dimensional features for representing the correlation relationship among the audio features, virtual object features and the emotional features.

[0096] Based on the above, it can be known that the obtained text features include audio features, as well as virtual object features and emotional features used to constrain audio features. In other words, the virtual object features and emotional features are fused into the audio features. The fusion method can be feature splicing, feature weighting, or feature weighted splicing. The following examples illustrate different feature processing methods.

[0097] In feature concatenation, different features are connected together in a certain order to form a larger feature vector. Concatenate the vectors of audio features, virtual object features, and emotional features: x = [audio features, virtual object features, emotional features]. For example, if there are two feature vectors x1 = [1,2,3] and x2 = [4,5,6], then concatenate them together to get x = [1,2,3,4,5,6].

[0098] In feature weighting, different weights are assigned to different features so that they can be weighted averaged or combined according to their importance. These weights can be set manually or automatically learned by a learning algorithm. For example, for a vector x = [x1, x2, x3] with three features, different weights can be applied to each feature to obtain a weighted feature vector y = w1x1 + w2x2 + w3*x3, where w1, w2, and w3 are the weights of the features respectively.

[0099] Feature weighted concatenation combines feature concatenation and feature weighting. First, weight different features, and then concatenate the weighted features together to form a new feature vector. For example, for a vector with two features x1 = [1, 2, 3] and x2 = [4, 5, 6], and weights w1 = 0.7 and w2 = 0.3 are set for these two features respectively, the weighted and concatenated feature vector y = w1x1 + w2x2 = [1.8, 2.7, 3.6] can be calculated.

[0100] In practical applications, it is necessary to select an appropriate feature fusion method based on the specific situation. Feature concatenation is suitable for situations where there is no obvious correlation structure between features, while feature weighting can control the influence of features at a finer granularity. Feature weighted concatenation combines the advantages of both methods and can simultaneously consider the correlation between features and the weights of different features.

[0101] The mapping module converts the text feature vector into a low-dimensional encoding vector through an encoder, and assumes that these encoding vectors follow a parameterized Gaussian distribution. Use the mean μ and variance σ learned during training to generate specific low-dimensional feature samples. For example, using the variational inference method, a random noise vector ∈ can be sampled from a standard normal distribution (mean 0, variance 1) by utilizing the reparameterization technique, and then combined with μ and σ to obtain the sampled low-dimensional feature sample. In this way, text features can be modeled and represented more compactly and relevantly in a low-dimensional space, providing valuable input for subsequent tasks and applications.

[0102] In one or more embodiments of the present application, the step of inputting the text features into a time prediction module for processing to obtain time prediction information includes: inputting the text features into the time prediction module to obtain prediction information in low-dimensional features for characterizing phoneme duration, so as to align the speech signal with the lip shape signal based on the prediction information.

[0103] Generally speaking, the correspondence between phonemes and mouth shapes is complex and varied. Multiple phonemes may correspond to one mouth shape, or one phoneme may correspond to one mouth shape. Therefore, when generating lip shape animation, in order to ensure that the obtained lip shape animation has good consistency with the played speech, it is necessary to use the time prediction module to predict the phoneme duration.

[0104] It should be noted that in the present application, low-dimensional features are input into the time prediction model. Since the low-dimensional features not only include features that represent phonemes, but also include features that represent emotions and virtual objects, the predicted time length is more in line with the emotional state and the personality or role characteristics of the virtual object, and can be consistent with the time length of the lip shape signal. At the same time, the generated speech and lip shape are more natural and more in line with the constraints of the corresponding text and character characteristics and emotional characteristics.

[0105] In one or more embodiments of the present application, the normalization processing is performed based on the virtual object code, the emotion code, the low-dimensional feature and the time prediction information to obtain the comprehensive feature, including:

[0106] Inputting the virtual object code and the emotion code into at least one flow layer in a flow module;

[0107] inputting the time prediction information into a designated flow layer;

[0108] In the flow module, the low-dimensional features, the virtual object code, and the emotion code are normalized to obtain comprehensive features that can express the emotions of the virtual object.

[0109] When normalization is performed, comprehensive features that can express the emotions of the virtual object are obtained.

[0110] Virtual object coding and emotion coding are vector representations of virtual objects and emotions. The flow layer is a structure used in neural networks to process sequence data, which can capture the time dependency and dynamic laws in the input data. Virtual object coding and emotion coding are input into at least one flow layer in the flow module to learn and capture the time dependency and dynamic laws in the input data. In order to better reflect the virtual object coding and emotion coding in the comprehensive features, virtual object coding and emotion coding are often input into different flow layers multiple times.

[0111] The time prediction information includes the predicted duration information of speech and lip shape. This duration information is input into a specific flow layer and may be used to help the model learn patterns and rules related to duration. By taking duration information as input, the flow layer can better adapt to outputs of different durations.

[0112] In the flow module, low-dimensional features are combined with virtual object encoding and emotion encoding and normalized. This means mapping the value ranges of different features to a uniform interval or distribution. Normalization makes the features comparable and balanced, which helps the comprehensive features better express the emotions of virtual objects. In the flow module, by processing virtual object encoding, emotion encoding, and time prediction information, a comprehensive feature is obtained that can better express the emotions of virtual objects and capture the time dependencies and dynamic evolution in the input data.

[0113] For example, if Figure 3 A schematic diagram of a process for adding virtual objects and emotion coding features provided in an embodiment of the present application. Figure 3 As can be seen in , constraints (virtual object encoding, emotion encoding) can be added to affect the distribution of the results generated by the variational autoencoder (VAE). Corresponding to this solution Figure 3 As shown, two constraints, a virtual character and the character's emotion, are added. By inputting the character ID and the emotion ID, speech and lip shape with emotion that conform to the characteristics of the character are generated. Specifically, the character ID and the emotion ID are encoded to the specified dimension through the Embedding layer, and then in the reasoning process, they are combined with the time prediction module, the flow layer flow module, and each decoding module decoder, respectively, to well integrate the constraints such as the character and emotion. Specifically, taking the flow layer flow-layer as an example, the Add method is used to add to the encoding features of the flow layer that has been reduced in dimension by projection, and the method of adding each layer is adopted so that the added constraints such as the character and emotion are more obvious.

[0114] In one or more embodiments of the present application, decoding the comprehensive feature to obtain a speech signal and a matching lip signal corresponding to the text information includes:

[0115] inputting the comprehensive features into a first decoding model and a second decoding model respectively;

[0116] Obtaining a lip shape weight coefficient output by the first decoding model and a speech signal output by the second decoding model;

[0117] A lip shape signal matching the speech signal is obtained based on the lip shape weight coefficient.

[0118] In practical applications, the comprehensive features are input into the first decoding model and the second decoding model respectively, wherein the first decoding model is used to output the lip shape weight coefficient, and the second model is used to output the speech signal.

[0119] The first decoding model generates lip shape weight coefficients based on the comprehensive features of the input. These weight coefficients reflect which lip shape should be used to produce the corresponding sound at different times. The lip shape weight coefficient determines the mouth shape at each time step in the speech signal.

[0120] The second decoding model generates a speech signal based on the comprehensive features of the input. This speech signal is obtained by learning the audio features and acoustic model through the model, and it can be used as a synthesis of sound.

[0121] After using the lip shape weight coefficient output by the first decoding model, a lip shape signal matching the speech signal output by the second decoding model can be generated according to the selected mouth shape. The lip shape signal is performed simultaneously with the sound, and it determines the dynamic changes of the mouth shape. Thus, the speech signal and the lip shape signal are synchronized and have better consistency. In this way, dynamic lip shape and corresponding speech signal can be generated simultaneously, thereby producing realistic lip shape animation effect.

[0122] For example, if Figure 5 A schematic diagram of lip shape processing provided in an embodiment of the present application.

[0123] The speech and lip shape generation are integrated into the same model, so that the model can output the speech and the corresponding lip shape at the same time. This can significantly improve the efficiency of lip shape animation generation, and because the speech and lip shape are obtained based on the same comprehensive features, they have better synchronization and consistency effects.

[0124] In terms of model structure, Figure 5As shown in the figure, the blendshape decoder module (that is, the lip shape decoding module) is designed first. The structure of the module is similar to that of the wav decoder module (that is, the speech decoding module). Through the design of network parameters, a gradual denoising and purification process similar to the diffusion stage of the SD model is achieved. Specifically, through the design of network parameters, the module gradually removes noise or irrelevant components during the lip shape decoding process, and extracts key facial expression features to generate more accurate and realistic lip shape animations. Similar to the operation of the SD model, the generation process of facial animation can be controlled in the Blendshape Decoder module by operating and adjusting the weight coefficients. Gradually reducing unnecessary deformation or noise information makes the generated lip shape animation higher in quality and more in line with expectations. This process is similar to the idea of ​​gradually removing noise or enhancing signals in the SD model.

[0125] In addition, in order to further improve the performance of facial / lip animation generation, a lip rendering module can be added to simulate the process of synthesizing a 3D face mesh from blendshape, which is implemented with a small VAE model. Compared with the weight coefficient, the synthesized face can more intuitively reflect the generation effect, so for the lip part, the weight coefficient loss and the face reconstruction loss are combined; specifically, for the weight coefficient loss, the difference between the generated weight coefficient and the target weight coefficient can be calculated and used as one of the objective functions of the optimization model. For the face reconstruction loss, the generated face mesh can be compared with the real face mesh using a deformation model or other methods, the difference between them can be calculated, and used as another optimization target. By combining these two loss functions, the quality of lip animation and the accuracy of facial shape can be optimized at the same time. When processing data, the time length and time step of the speech and lip sequence are aligned by setting the sampling frequency, spectrum frequency, and number of lip frames in conjunction with the model parameters. It is important to ensure that the length of the speech and lip sequence is aligned. This means that before model training, the speech and lip data need to be preprocessed, which can be achieved by setting the appropriate sampling frequency, spectrum frequency, and number of lip frames for lip animation. By aligning these parameters, we can ensure that the speech signal and the corresponding lip sequence are input into the model in a related way.

[0126] By combining the Blendshape Decoder module with the lip rendering module, the performance of the generated facial / lip animation can be improved. At the same time, adjusting the sequence length to maintain alignment during data processing helps to ensure the consistency of the generated lip and speech signals. These improvements can further improve the quality and realism of the generated lip animation.

[0127] Based on the same idea, the present application embodiment also provides a model training method. Figure 6A flow chart of the model training method provided in the embodiment of the present application. Figure 6 As can be seen in the figure, the method comprises the following steps:

[0128] Step 601: Obtain training samples consisting of text information, virtual object identifiers, and emotion identifiers.

[0129] Step 602: Input the training sample into the model to be trained so that the trained machine learning model has encoding and decoding capabilities. The encoding model in the machine learning model is used to output comprehensive features for characterizing the common features of speech and lip shape according to the input text information, virtual object identifier, and emotion identifier; the decoding model in the machine learning model is used to decode the speech signal and the matching lip shape signal according to the comprehensive features.

[0130] In practical applications, training samples consist of text information, virtual object identifiers, emotion identifiers, and expected speech signals (such as audio sequences) and lip signals (such as lip sequences). These samples may include text data related to speech and lip shape, identifiers of virtual objects (such as the speaker's ID), and identifiers related to emotions (such as the speaker's emotional state). It should be noted that in this solution, text information, virtual objects, and emotions are comprehensively utilized during training so that the trained machine learning model can directly convert speech signals and lip signals from text without the need for intermediate conversion, thereby generating lip animations with better matching. During the training process, the training samples are input into the model to be trained. This model is designed as a machine learning model, which includes an encoding model and a decoding model to achieve encoding and decoding capabilities.

[0131] The role of the encoding model is to generate a comprehensive feature based on the input text information, virtual object identifiers and emotion identifiers. The comprehensive feature represents the common features between speech and lip shape. Therefore, when training the encoding model, the encoding model needs to capture the correlation between speech and lip shape, that is, the common features, by learning the mapping relationship from input features to comprehensive features. This allows the subsequent decoding to output both speech signals and lip shape signals at the same time.

[0132] The role of the decoding model is to decode and generate the corresponding speech signal and the corresponding lip shape signal according to the given comprehensive features. The decoding model learns the reconstruction process from the comprehensive features to the speech signal and the lip shape signal, so that the speech signal and the lip shape signal that match it can be generated according to the given comprehensive features. By training this machine learning model, its encoding model can extract the comprehensive features that characterize the common features of speech and lip shape from the input features, and the decoding model can use these comprehensive features to generate the corresponding speech signal and lip shape signal. The first model obtained through the above training has the ability to encode and decode, and can convert the given text information, virtual object identifier and emotion identifier into the corresponding speech and lip shape at the same time. During use, the user can directly input text information to obtain the lip shape animation. In the process of converting text information into lip shape animation, there is no need to convert to the intermediate state (speech state), which can effectively improve the efficiency of lip shape animation generation.

[0133] In one embodiment of the present application, the model to be trained includes a lip rendering model.

[0134] The training method of the model to be trained includes: inputting the training comprehensive features obtained during the training process into the decoding model to be trained to obtain a first loss corresponding to the lip decoding weight coefficient; and inputting the lip decoding weight coefficient into the lip rendering model to be trained to obtain a second loss; and iteratively optimizing the model to be trained based on the first loss and the second loss.

[0135] The comprehensive features obtained during the training process are input into the decoding model to be trained to generate the lip-decoding weight coefficients. The decoding model learns a function that maps the comprehensive features to the lip-decoding weight coefficients. The lip-decoding weight coefficients are input into the lip-rendering model to be trained to generate a synthetic 3D facial mesh. The lip-rendering model uses the lip-decoding weight coefficients to generate realistic facial shapes.

[0136] The first loss of the lip-sync decoding weight coefficients, namely the weight coefficient loss, is calculated. This loss measures the difference between the generated weight coefficients and the target weight coefficients, thereby guiding the model to optimize the quality of the generated lip animation. The facial reconstruction loss of the lip-sync decoding weight coefficients is calculated to measure the difference between the generated facial mesh and the real facial mesh. This loss measures the accuracy and realism of the facial shape.

[0137] The first loss and the second loss are combined together, and the lip rendering model is iteratively optimized. Through the back propagation algorithm, the loss is passed backward and the parameters of the model are updated to minimize the weight coefficient loss and the face reconstruction loss. In this way, the model can improve the quality of the lip animation and the accuracy of the facial shape through iterative optimization. Through this training process, the lip rendering model can learn to map the input comprehensive features into the lip decoding weight coefficient with a realistic facial shape. In this embodiment, the lip rendering model to be trained can also be pre-trained separately. Through separate training, not only the main model training time is saved, but also the training of the main model will be more stable after the lip rendering separate training converges. By combining the weight coefficient loss and the face reconstruction loss, the model can generate a lip animation that conforms to the input text and the virtual object identifier, and can ensure that the generated facial mesh is closer to the real sample. Continuously optimizing the lip rendering model can improve the quality and realism of the generated results.

[0138] For example, to train the lip rendering module, we use a small variational autoencoder (VAE) to simulate the process of synthesizing 3D facial meshes from Blendshape. In the lip part, in order to better optimize the generated results, we combine the weight coefficient loss and the facial reconstruction loss. The weight coefficient loss is used to control the quality and accuracy of the lip animation. By adjusting the weight coefficient, the generated lip shape can be changed. The facial reconstruction loss is used to measure the difference between the generated 3D facial mesh and the real facial mesh, so that the generated facial shape is closer to the real sample. When training the lip rendering module, first prepare training samples containing lip parameters and corresponding facial mesh data. Then, these samples are input into a small VAE model, where the encoder part maps the lip parameters to the distribution of the latent space and generates the corresponding latent features. The decoder part decodes the latent features into the generated facial mesh. This process can be regarded as the encoding and decoding process from lip parameters to facial mesh. During the training process, we use the weight coefficient loss and the facial reconstruction loss to optimize the performance of the model. The weight coefficient loss calculates the difference between the generated weight coefficient and the target weight coefficient and uses it as part of the optimization model. The facial reconstruction loss measures the reconstruction error by comparing the difference between the generated facial mesh and the real facial mesh, and uses it as another optimization objective. Combining these two loss functions can optimize both the quality of lip animation and the accuracy of facial shape. By training the lip rendering module, we hope that it can generate realistic facial meshes that can accurately reflect the generated effects of text and virtual object labels, while ensuring the quality of lip animation through weight coefficient loss. In this way, the generated facial animation will be more intuitive, realistic, and better match the input text and virtual object labels.

[0139] In one or more embodiments of the present application, the model to be trained further includes a discriminator;

[0140] The training method of the model to be trained includes: inputting the trained speech signal and lip shape signal into corresponding discriminators respectively to obtain a classification prediction loss that characterizes whether it is a real signal and a feature map loss extracted based on the signal; and iteratively optimizing the model to be trained using the classification prediction loss and the feature map loss.

[0141] For example, in model training, in addition to learning the synthesis loss of speech and lip shape, in order to further improve the model training effect, adversarial learning can be added to make the distribution of the generator and the real data close, increase the robustness of the model, and make the generated speech and lip shape more realistic and natural. Set a discriminator for the generated speech signal and lip shape signal respectively. Figure 4 This is a schematic diagram of the discriminator training process provided in the embodiment of the present application. Figure 4 As shown in the figure, prediction and true label are usually used as input when training the discriminator. The output of the discriminator is divided into two parts: the classification prediction of whether the input signal is a real signal and the feature map extracted from the signal, and the adversarial loss and feature map loss are obtained respectively.

[0142] Adding a discriminator can improve the realism of the generated speech and lip shape. Specifically, by introducing a discriminator, the generator model can be trained to imitate the distribution of real data, thereby generating more realistic and natural speech and lip shape signals. The discriminator can assist the generator model in learning the characteristics of real data and help the model generate more and more realistic results. In addition, the robustness and generalization of the model are increased, and the learning and expression capabilities of the model are improved. The discriminator can provide feedback on the quality and realism of the generated signal. The generator can improve the output results by optimizing the generated signal to better deceive the discriminator. This prompts the generator model to learn higher-level speech and lip shape representations, improving the learning and expression capabilities of the model. Therefore, by introducing a discriminator and adversarial learning, the lip rendering model can further improve the training effect and generate more realistic and natural speech and lip shape signals. The existence of the discriminator can help the model learn the characteristics of real data, and improve the robustness, generalization and learning ability of the model, making the generated results more realistic and accurate.

[0143] In one or more embodiments of the present application, the model to be trained includes a time prediction module to be trained; the training method of the time prediction module to be trained includes: obtaining training comprehensive features and training phoneme sequences; establishing a corresponding relationship between the training comprehensive features and the training phoneme sequences; inputting the training comprehensive features and the training phoneme sequences with the corresponding relationship into the time prediction module to be trained, so that the time prediction module obtained by training outputs prediction information representing the phoneme duration.

[0144] For example, the time series module outputs the sequence length (integer) of each phoneme corresponding to the latent space Z. Since the subsequent decoder has a fixed magnification factor for the length of the Z sequence, it can be approximately considered that the time series prediction module directly outputs the duration of each phoneme. During training, the Monotonic Alignment Search (MAS) method is used to dynamically program the lengths of the z sequence elements and the phoneme sequence elements one by one (i.e., the z sequence elements are assigned to each phoneme in turn). During inference, we can adjust the magnification factor (the default is 1) to scale the Z sequence to achieve the effect of speeding up or slowing down the audio.

[0145] For ease of understanding, the following will illustrate the lip animation generation process through a specific embodiment. Figure 7 This is a schematic diagram of the lip animation generation process provided in the embodiment of the present application. The details are as follows:

[0146] The overall process of the program is:

[0147] 1. Input: text and corresponding character id and emotion id.

[0148] The text needs to be converted into corresponding phonemes. There are many phoneme conversion methods depending on the language. Taking Chinese as an example, the process is: word segmentation, conversion to pinyin, and conversion to pinyin phonemes. The character id is the character number to be generated (integer value, that is, the character serial number included in the model, which can also be left blank and set to None to get a general voice); the emotion id is the emotion number (integer value, representing one of the emotions such as calm, angry, happy, sad, etc., which can also be set to None to get a general emotion).

[0149] 2. Encoding process of the Encoder part:

[0150] 1) First, the phoneme sequence and character and emotion codes are input into [text encoder] for text feature encoding.

[0151] 2) Input the text features into the duration predictor module to obtain the time prediction information (i.e., the audio and lip length); and simultaneously input them into the projection module to further combine the text features to obtain the mean μ and variance σ of the fitted Z distribution.

[0152] 3) Input the character emotion encoding, Z distribution estimation and time information into the normalized flow module [flow] to improve the feature expression ability and integrate them to obtain the final feature latent Z.

[0153] 3. Decoding Decoder part:

[0154] The encoded temporal comprehensive features Z are input into 1) [Blendshape Decoder] to obtain the weight coefficients of each lip shape blendshape template; 2) [Wav Decoder] to obtain the final speech signal. For the lip shape, the template weight coefficients are combined and rendered to obtain the final lip shape animation.

[0155] Based on the same idea, the embodiment of the present application also provides a lip animation generation device. Figure 8 A schematic diagram of the structure of a lip animation generation device provided in an embodiment of the present application. Figure 8 As can be seen, the device includes:

[0156] The acquisition module 81 is used to acquire text information, virtual object identifiers, and emotion identifiers.

[0157] The conversion module 82 is used to convert the text information into a phoneme sequence, convert the virtual object identifier into a virtual object code, and convert the emotion identifier into an emotion code.

[0158] The encoding module 83 is used to input the phoneme sequence, the virtual object code, and the emotion code into the encoding model for feature combination to obtain comprehensive features for characterizing the common features of speech and lip shape.

[0159] The decoding module 84 is used to decode the comprehensive features to obtain the voice signal and the matching lip shape signal corresponding to the text information, so as to generate the lip shape animation using the voice signal and the lip shape signal.

[0160] An encoding module 83, used for inputting the phoneme sequence, the virtual object code and the emotion code into a text feature model to obtain text features;

[0161] Inputting the text features into a mapping module for feature combination to obtain low-dimensional features;

[0162] Inputting the text features into a time prediction module for processing to obtain time prediction information;

[0163] Normalization processing is performed based on the virtual object code, the emotion code, the low-dimensional feature and the time prediction information to obtain the comprehensive feature.

[0164] The encoding module 83 is used to input the text features including audio features, virtual object features representing constraint conditions and emotional features into the mapping module for feature splicing and / or feature weighting to obtain low-dimensional features for representing the correlation relationship between the audio features, virtual object features and the emotional features.

[0165] The encoding module 83 is used to input the text features into the time prediction module to obtain prediction information for representing the phoneme duration in the low-dimensional features, so as to align the speech signal with the lip signal based on the prediction information.

[0166] An encoding module 83, used for inputting the virtual object encoding and the emotion encoding into at least one flow layer in a flow module;

[0167] inputting the time prediction information into a designated flow layer;

[0168] In the flow module, the low-dimensional features, the virtual object code, and the emotion code are normalized to obtain comprehensive features that can express the emotions of the virtual object.

[0169] A decoding module 84, used to input the comprehensive features into a first decoding model and a second decoding model respectively;

[0170] Obtaining a lip shape weight coefficient output by the first decoding model and a speech signal output by the second decoding model;

[0171] A lip shape signal matching the speech signal is obtained based on the lip shape weight coefficient.

[0172] Based on the same idea, the present application embodiment also provides a model training device. Fig. 9 A schematic diagram of the structure of a model training device provided in an embodiment of the present application. Fig. 9 As can be seen, the device includes:

[0173] The acquisition module 91 acquires a training sample consisting of text information, a virtual object identifier, and an emotion identifier.

[0174] A training module 92, used for inputting the training samples into the model to be trained so that the trained machine learning model has encoding and decoding capabilities;

[0175] Among them, the encoding model in the machine learning model is used to output comprehensive features for characterizing the common characteristics of speech and lip shape based on the input text information, virtual object identifier, and emotion identifier; the decoding model in the machine learning model is used to decode the speech signal and the matching lip shape signal based on the comprehensive features.

[0176] Optionally, the model to be trained includes a lip rendering model to be trained.

[0177] A training module 92, used for inputting the training comprehensive features obtained in the training process into the decoding model to be trained, and obtaining a first loss corresponding to the lip decoding weight coefficient;

[0178] and inputting the lip decoding weight coefficient into the lip rendering model to be trained to obtain a second loss;

[0179] The model to be trained is iteratively optimized based on the first loss and the second loss.

[0180] Optionally, the model to be trained also includes a discriminator.

[0181] The training module 92 is used to input the trained speech signal and lip shape signal into the corresponding discriminator to obtain the classification prediction loss representing whether it is a real signal and the feature map loss extracted based on the signal;

[0182] The model to be trained is iteratively optimized using the classification prediction loss and the feature map loss.

[0183] Optionally, the model to be trained includes a time prediction module to be trained;

[0184] A training module 92, used to obtain training comprehensive features and training phoneme sequences;

[0185] Establishing a corresponding relationship between the training comprehensive feature and the training phoneme sequence;

[0186] The training comprehensive features and the training phoneme sequence having a corresponding relationship are input into the time prediction module to be trained, so that the trained time prediction module outputs prediction information representing the phoneme duration.

[0187] Based on the above embodiment, text information, virtual object identifiers, and emotion identifiers are obtained; the text information is converted into a phoneme sequence, the virtual object identifier is converted into a virtual object code, and the emotion identifier is converted into an emotion code; the phoneme sequence, the virtual object code, and the emotion code are input into a coding model for feature combination to obtain a comprehensive feature for characterizing the common features of speech and lip shape; the comprehensive feature is decoded to obtain a speech signal and a matching lip shape signal corresponding to the text information, so as to generate a lip shape animation using the speech signal and the lip shape signal. Through the above scheme, the input text information, virtual object identifier, and emotion identifier are converted and processed by the coding model to obtain a comprehensive feature that can be used to characterize the common features of speech and lip shape, without the cumbersome process of converting text into speech and then converting to obtain lip shape data. After obtaining the comprehensive feature, the comprehensive feature is decoded to obtain a speech signal and a corresponding lip shape signal corresponding to the same text information, and then a lip shape animation containing speech is generated at the same time, thereby improving the work efficiency of directly generating lip shape animation from text information, and the obtained speech and lip shape animation have good consistency.

[0188] In one possible design, the above Figure 8 The lip animation generating device shown and Fig. 9 The structure of the model training device shown can be implemented as an electronic device. Fig.10 As shown, the electronic device may include: a processor 101 and a memory 102. The memory 102 stores executable code, and when the executable code is executed by the processor 101, the processor 101 can at least implement the visual effect enhancement method provided in the above embodiment. The structure of the electronic device may also include a communication interface 103 for communicating with other devices or communication networks.

[0189] In addition, an embodiment of the present invention provides a non-transitory machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor in a server or a computer, the processor executes the operations provided in the above embodiments. Figure 1 and Figure 6 The corresponding method.

[0190] The device embodiments described above are merely illustrative, wherein the modules described as separate components may or may not be physically separated. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. Those of ordinary skill in the art may understand and implement the present invention without creative effort.

[0191] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by adding a necessary general hardware platform, and of course can also be implemented by combining hardware and software. Based on such an understanding, the above technical solution can essentially or contribute to the part in the form of a computer product, and the present invention can be implemented in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0192] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating lip animation, characterized in that: The method comprises: Obtaining text information, virtual object identifiers, and emotion identifiers; Converting the text information into a phoneme sequence, converting the virtual object identifier into a virtual object code, and converting the emotion identifier into an emotion code; Inputting the phoneme sequence, the virtual object code, and the emotion code into a coding model for feature combination to obtain a comprehensive feature for characterizing common features of speech and lip shape; The comprehensive features are decoded to obtain a voice signal and a matching lip shape signal corresponding to the text information, so as to generate a lip shape animation using the voice signal and the lip shape signal.

2. The method according to claim 1, characterized in that The step of inputting the phoneme sequence, the virtual object code, and the emotion code into a coding model for feature combination to obtain a comprehensive feature for characterizing common features of speech and lip shape includes: Inputting the phoneme sequence, the virtual object code and the emotion code into a text feature model to obtain text features; Inputting the text features into a mapping module for feature combination to obtain low-dimensional features; Inputting the text features into a time prediction module for processing to obtain time prediction information; Normalization processing is performed based on the virtual object code, the emotion code, the low-dimensional feature and the time prediction information to obtain the comprehensive feature.

3. The method according to claim 2, characterized in that The step of inputting the text features into a mapping module for feature combination to obtain low-dimensional features includes: The text features including audio features, virtual object features representing constraint conditions and emotional features are input into a mapping module for feature concatenation and / or feature weighting to obtain low-dimensional features for representing the correlation relationship among the audio features, virtual object features and the emotional features.

4. The method according to claim 2, characterized in that: The step of inputting the text features into a time prediction module for processing to obtain time prediction information includes: The text features are input into the time prediction module to obtain prediction information for representing the phoneme duration in the low-dimensional features, so as to align the speech signal with the lip signal based on the prediction information.

5. The method according to claim 2, characterized in that: The normalization process based on the virtual object code, the emotion code, the low-dimensional feature and the time prediction information to obtain the comprehensive feature includes: Inputting the virtual object code and the emotion code into at least one flow layer in a flow module; inputting the time prediction information into a designated flow layer; In the flow module, the low-dimensional features, the virtual object code, and the emotion code are normalized to obtain comprehensive features that can express the emotions of the virtual object.

6. The method according to claim 1, characterized in that The decoding of the comprehensive features to obtain a speech signal corresponding to the text information and a matching lip signal includes: inputting the comprehensive features into a first decoding model and a second decoding model respectively; Obtaining a lip shape weight coefficient output by the first decoding model and a speech signal output by the second decoding model; A lip shape signal matching the speech signal is obtained based on the lip shape weight coefficient.

7. A model training method, characterized in that: The method comprises: Obtaining training samples consisting of text information, virtual object identifiers, and emotion identifiers; Inputting the training samples into the model to be trained so that the trained machine learning model has encoding and decoding capabilities; Among them, the encoding model in the machine learning model is used to output comprehensive features for characterizing the common characteristics of speech and lip shape based on the input text information, virtual object identifier, and emotion identifier; the decoding model in the machine learning model is used to decode the speech signal and the matching lip shape signal based on the comprehensive features.

8. The method according to claim 7, characterized in that The model to be trained includes a lip rendering model to be trained; The training method of the model to be trained includes: Inputting the comprehensive training features obtained during the training process into the decoding model to be trained to obtain a first loss corresponding to the lip decoding weight coefficient; and inputting the lip decoding weight coefficient into the lip rendering model to be trained to obtain a second loss; The model to be trained is iteratively optimized based on the first loss and the second loss.

9. The method according to claim 7, characterized in that: The model to be trained also includes a discriminator; The training method of the model to be trained includes: The trained speech signal and lip shape signal are input into the corresponding discriminator respectively to obtain the classification prediction loss that represents whether it is a real signal and the feature map loss extracted based on the signal; The model to be trained is iteratively optimized using the classification prediction loss and the feature map loss.

10. The method according to claim 7, characterized in that The model to be trained includes a time prediction module to be trained; The training method of the time prediction module to be trained includes: Obtaining training comprehensive features and training phoneme sequences; Establishing a corresponding relationship between the training comprehensive feature and the training phoneme sequence; The training comprehensive features and the training phoneme sequence having a corresponding relationship are input into the time prediction module to be trained, so that the trained time prediction module outputs prediction information representing the phoneme duration.

11. A lip animation generating device, characterized in that: The device comprises: An acquisition module, used to acquire text information, virtual object identifiers, and emotion identifiers; A conversion module, used to convert the text information into a phoneme sequence, convert the virtual object identifier into a virtual object code, and convert the emotion identifier into an emotion code; An encoding module, used for inputting the phoneme sequence, the virtual object code, and the emotion code into an encoding model for feature combination to obtain a comprehensive feature for characterizing common features of speech and lip shape; A decoding module is used to decode the comprehensive features to obtain a voice signal and a matching lip shape signal corresponding to the text information, so as to generate a lip shape animation using the voice signal and the lip shape signal.

12. An electronic device comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 10.

13. A computer-readable medium having stored thereon at least one instruction, at least one program, code set or instruction set, wherein the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the method according to any one of claims 1 to 10.

Images