Mouth shape animation generation method and device, mouth shape animation model training method and device, equipment and medium

Abstract

The invention discloses a mouth shape animation generation method and device, a model training method and device, equipment and a medium. The method comprises the following steps: acquiring an original mouth shape animation weight; inputting the original mouth shape animation weight into a first mouth shape animation model, and performing pixel difference correction on the original mouth shape animation weight by using the first mouth shape animation model to obtain a reconstructed mouth shape animation weight; inputting the original mouth shape animation weight into the second mouth shape animation model to obtain a key point shape value corresponding to the key point coordinate; and based on the corresponding relationship between the key point coordinates and the reconstructed mouth shape animation, generating a target mouth shape animation after the original mouth shape animation is corrected by fusing the key point shape value and the reconstructed mouth shape animation weight. According to the method, when the mouth shape animation is generated, the difference between the synthesized mouth shape and the real mouth shape is compared in the pixel dimension, so that the generated mouth shape animation is more reasonable, and meanwhile, the key point regression network is combined to apply stronger constraint to the mouth area, so that the synthesis effect is closer to the real mouth shape.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of Internet technology, and in particular to lip animation generation, model training methods, devices, equipment and media. Background Art

[0002] In a virtual interactive scene, users can watch the various actions of virtual targets in the scene, and some users can also interact with virtual objects to gain a greater sense of participation.

[0003] Among them, lip animation is an important part of the animation production process. When generating lip animation, some methods comprehensively determine the lip animation frames corresponding to the current text or voice based on the correspondence between phonemes and lip weights. However, in practical applications, there is a significant gap between the lip animation generated based on lip weights and the actual expectations. In order to narrow the gap, professionals are required to adjust the lip animation frames, which is time-consuming and laborious. Therefore, a solution is needed to quickly generate lip animation that meets actual needs. Summary of the invention

[0004] The embodiments of the present invention provide a method, device, equipment and medium for generating and training lip animation and a model, so as to improve the consistency effect between lip animation and actual phonemes.

[0005] In a first aspect, an embodiment of the present invention provides a method for generating a lip animation, the method comprising:

[0006] Obtaining an original lip animation weight; wherein the original lip animation weight is determined based on phonemes;

[0007] Inputting the original lip animation weight into a first lip animation model, and using the first lip animation model to perform pixel difference correction on the original lip animation weight to obtain a reconstructed lip animation weight;

[0008] Inputting the original lip animation weight into the second lip animation model to obtain key point shape values ​​corresponding to the key point coordinates;

[0009] Based on the correspondence between the key point coordinates and the reconstructed lip animation, the key point shape values ​​and the reconstructed lip animation weights are fused to generate a target lip animation after the original lip animation is corrected.

[0010] Optionally, based on the correspondence between the key point coordinates and the reconstructed lip animation, generating a target lip animation after the original lip animation is corrected by fusing the key point shape values ​​with the reconstructed lip animation weights, comprising:

[0011] According to the corresponding relationship, determining a shape offset of the key point coordinates relative to the basic blend shape by subtracting the key point shape value from the neutral shape value of the basic blend shape;

[0012] Determining a weight correction value based on the reconstructed lip animation weight and the shape offset;

[0013] The target lip animation is generated by modifying the basic animation corresponding to the basic blend shape using the weight correction value.

[0014] Optionally, the original lip animation weight is input into a first lip animation model, and the original lip animation weight is corrected for pixel differences using the first lip animation model to obtain a reconstructed lip animation weight, including:

[0015] Performing feature extraction on the original lip animation weight to obtain a first original image feature;

[0016] After inputting the first original image feature into the first lip animation model, generating a predicted feature map;

[0017] Based on the pixel difference between the predicted feature map and the real image, feature correction is performed to obtain the corresponding corrected reconstructed lip animation weight.

[0018] Optionally, after inputting the first original image feature into the first lip animation model, generating a predicted feature map includes:

[0019] Inputting the first original image feature into the encoding model in the first lip animation model to generate a predicted image;

[0020] The predicted image is input into a decoding model for downsampling processing to obtain the predicted feature map.

[0021] Optionally, the performing feature correction based on the pixel difference between the feature map and the real image to obtain the corresponding corrected predicted reconstructed lip animation weight includes:

[0022] Inputting the feature map into a first regression network in the first lip animation model, performing difference correction on the feature map, and obtaining a corrected feature map;

[0023] The modified feature map is converted into the reconstructed lip animation weight having the same dimension as the original lip animation weight.

[0024] Optionally, the first regression network is trained based on actual image pixel comparison results; wherein the training method of the first regression network includes:

[0025] Acquire training samples, wherein the training samples include: original lip animation weight samples, predicted image samples and key point shape value samples;

[0026] The training samples are input into the first model to be trained, and the first regression network is obtained by training using a loss function that characterizes the pixel difference between the predicted image and the predicted image sample.

[0027] Optionally, the second lip animation model includes a second regression network;

[0028] The step of inputting the original lip animation weight into the second lip animation model to obtain key point shape values ​​corresponding to key point coordinates includes:

[0029] Based on the preset key point coordinates, the original lip animation weights are input into the second lip animation model for conversion processing to obtain key point shape values ​​corresponding to the key point coordinates.

[0030] In a second aspect, an embodiment of the present invention provides a model training method, the method comprising:

[0031] Acquire training samples for training the first model to be trained and the second model to be trained; wherein the training samples include: original lip animation weight samples, predicted image samples and key point shape value samples;

[0032] Using the training samples to train the first model to be trained to obtain a first lip animation model, wherein the first lip animation model is used to modify the original lip animation weight to obtain a reconstructed lip animation weight;

[0033] The second model to be trained is trained using the training samples to obtain a second lip animation model, wherein the second lip animation model is used to determine the key point coordinates corresponding to the reconstructed lip animation weights respectively; so that after the model training is completed, the key point coordinates and the reconstructed lip animation weights are fused to generate a target lip animation after the original lip animation is corrected.

[0034] In a third aspect, an embodiment of the present invention provides a lip animation generation device, the device comprising:

[0035] The acquisition module is used to obtain the original lip animation weight;

[0036] A first input module, configured to input the original lip animation weight into a first lip animation model, and use the first lip animation model to perform pixel difference correction on the original lip animation weight to obtain a reconstructed lip animation weight;

[0037] A second input module, used for inputting the original lip animation weight into a second lip animation model to obtain key point shape values ​​corresponding to key point coordinates;

[0038] A generation module is used to generate a target lip animation after the original lip animation is corrected by fusing the key point shape value and the reconstructed lip animation weight based on the correspondence between the key point coordinates and the reconstructed lip animation.

[0039] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor to implement the lip animation generation method according to the first aspect.

[0040] In a fourth aspect, an embodiment of the present invention further provides a computer-readable medium having stored thereon at least one instruction, at least one program, a code set or an instruction set, wherein the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by a processor to implement the lip animation generation method according to the first aspect.

[0041] In an embodiment of the present invention, the weight of the original lip animation is obtained; wherein the weight of the original lip animation is determined based on phonemes; the weight of the original lip animation is input into a first lip animation model, and the weight of the original lip animation is corrected for pixel differences using the first lip animation model to obtain the weight of the reconstructed lip animation; the weight of the original lip animation is input into a second lip animation model to obtain the key point shape value corresponding to the key point coordinates; based on the correspondence between the key point coordinates and the reconstructed lip animation, the key point shape value and the reconstructed lip animation weight are fused to generate the target lip animation after the original lip animation is corrected. Through the above embodiment, the weight of the original lip animation obtained by the traditional method is corrected to obtain the weight of the reconstructed lip animation; at the same time, the key point shape value corresponding to the key point coordinates needs to be used to comprehensively generate the lip animation. Since the pixel dimension compares the difference between the synthetic lip shape and the real lip shape when generating the lip animation here, the generated lip animation is more reasonable. Moreover, based on the use of weight values ​​to generate lip-sync animation, the key point regression network is further combined to impose stronger constraints on the mouth area, making the synthesis effect of the mouth closer to the real lip-sync animation. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 A schematic diagram of a flow chart of a method for generating lip animation provided in an embodiment of the present application;

[0044] Figure 2 A schematic diagram of a basic blending shape for illustrating an embodiment of the present application;

[0045] Figure 3a A schematic diagram of a coding model for illustrating an embodiment of the present application;

[0046] Figure 3b A schematic diagram of a decoding model for illustrating an embodiment of the present application;

[0047] Figure 4 A schematic diagram of a first regression network is provided for an embodiment of the present application;

[0048] Figure 5 A schematic diagram of a second regression network is provided for an embodiment of the present application;

[0049] Figure 6 A flowchart of a model training method provided in an embodiment of the present application;

[0050] Figure 7 A schematic diagram of the structure of a lip animation generation device provided in an embodiment of the present application;

[0051] Figure 8 A schematic diagram of a model training device provided in an embodiment of the present application

[0052] Fig. 9 For Figure 7 A schematic diagram of the structure of an electronic device corresponding to the lip animation generation device provided in the embodiment;

[0053] Fig.10 A schematic diagram of the lip animation generation process for illustrating an embodiment of the present application;

[0054] Fig.11 A schematic diagram of a feature extractor provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two.

[0057] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0058] In addition, the step sequence in the following method embodiments is only an example and not a strict limitation.

[0059] In a virtual interactive scene, users can watch the various actions of virtual targets in the scene, and some users can also interact with virtual objects to gain a greater sense of participation.

[0060] Among them, lip animation is an important part of the animation production process. When generating lip animation, some methods comprehensively determine the lip animation frames corresponding to the current text or voice based on the correspondence between phonemes and lip weights. However, in practical applications, there is a significant gap between the lip animation generated based on lip weights and the actual expectations. In order to narrow the gap, professionals are required to adjust the lip animation frames, which is time-consuming and laborious. Therefore, a solution is needed to quickly generate lip animation that meets actual needs.

[0061] To facilitate understanding, specific embodiments will be used as examples below.

[0062] like Figure 1 The flowchart of the lip animation generation method provided in the embodiment of the present application is as follows. Figure 1 As you can see, the specific steps include the following:

[0063] Step 101: Obtaining the weight of the original lip animation; wherein the weight of the original lip animation is determined based on phonemes.

[0064] Step 102: inputting the original lip animation weight into a first lip animation model, and using the first lip animation model to perform pixel difference correction on the original lip animation weight to obtain a reconstructed lip animation weight.

[0065] Step 103: inputting the original lip animation weights into the second lip animation model to obtain key point shape values ​​corresponding to key point coordinates.

[0066] Step 104: Based on the correspondence between the key point coordinates and the reconstructed lip animation, the key point shape values ​​and the reconstructed lip animation weights are fused to generate a target lip animation after the original lip animation is corrected.

[0067] In practical applications, phoneme-based lip animation is a technology that converts phoneme information in text into corresponding facial animation or lip sequence. It can be used to synchronize lip shape with speech, so that virtual characters or animated characters can produce realistic lip movement when pronouncing. Phoneme is the smallest unit of speech in a language, representing different pronunciation units. In the process of phoneme-based lip animation, the following steps are generally included: Convert the input text into a corresponding phoneme sequence. This can be achieved by using text-to-phoneme conversion rules or by using existing speech recognition models. Map each phoneme to a corresponding lip shape BlendShape or facial animation. For example, an animation library containing original lip animations can be established, and the corresponding relationship between each phoneme and each original lip animation weight is stored in the library (usually a mapping table is established to manage the corresponding relationship). In addition, it can also be achieved by using a predefined mapping table of phoneme to original lip animation weights or by using machine learning techniques such as neural networks. By learning the mapping relationship between phonemes and original lip animation weights, the model can generate facial animations that match the input phoneme sequence. Render the lip animation corresponding to the phoneme into the final lip animation. This includes generating facial meshes based on mouth position and movement, and applying effects such as lighting and textures to improve realism. The generated lip animation is synthesized with other related image or video elements to produce the final lip animation effect. The key to phoneme-based lip animation is to establish an accurate mapping relationship from phonemes to mouth shapes, and to achieve realistic lip animation effects through lip rendering and animation synthesis technology. This technology has a wide range of applications in filmmaking, game development, virtual character design and other fields, and can give virtual characters more vivid and natural lip expression capabilities.

[0068] The first lip animation model mentioned here is a model for correcting the weight of the original lip animation. When the model is corrected, it is corrected based on the pixel difference. The first lip animation model can determine whether the pixel values ​​in the original lip animation obtained based on the phonemes are accurate, and can correct each pixel point with inaccurate pixel values. Because the user can see the final animated image, therefore, the correction effect achieved by correcting the pixel difference using the first lip animation model is more in line with the user's visual effect.

[0069] In order to obtain a lip animation that is more consistent with the phoneme content, further corrections need to be made to the more important positions. Specifically, some key points can be selected from the lip animation. These key points are the coordinate points near and around the mouth in the lip animation. Selecting some important key points to control the movements around the mouth can effectively affect the realism and expressiveness of the lip animation. The following are some common important key points: Mouth corner key points: The position and movement of the mouth corners are very important for the expression of the lip shape. The up and down movement of the mouth corners can express smiles, opening or closing the mouth, etc. Ensuring accurate capture and control of the movement of the mouth corners can make the lip animation more natural and vivid. Lip key points: The shape and movement of the lips directly affect the expression of the lip shape. Key points can be placed in the middle, sides and corners of the upper and lower lips to control the opening and closing of the lips, the position of the tongue, and the corresponding shape generated when pronouncing. Jaw key points: The movement of the jaw determines the degree of opening and closing of the mouth shape. Placing key points in the center and sides of the jaw can accurately capture the movement of the jaw and make the movement of the mouth shape more realistic. Teeth Keypoints: Select keypoints to control the position and shape of teeth, making the lip animation more realistic. The teeth can be shown or hidden according to the degree of mouth opening and closing and the pronunciation of specific phonemes.

[0070] The selection of these key points should be adjusted according to the specific situation, and you can refer to anatomical knowledge and actual lip-sync observations. By capturing and controlling these key points to achieve realistic lip-sync animation, you can improve the expressiveness and credibility of the animation.

[0071] After selecting the key points in the above manner, the weights of the original lip animation obtained based on the phonemes can be input into the trained second lip animation model, and then the second lip animation model calculates the key point shape values ​​corresponding to each key point. Since only a few important key points are corrected when correcting the vertices in the animation image, and there is no need to correct the vertices of the entire face, the purpose of correcting the lip animation can be achieved while effectively improving the correction efficiency.

[0072] Since the key points are the key points of the mouth, it is necessary to accurately correspond the coordinates of the key points of the mouth to the reconstructed lip animation, and then, according to the corresponding relationship, the weight of the reconstructed lip animation is corrected according to the key point shape value. This correction method is more accurate and efficient, and can achieve targeted correction. As a result, the target lip animation is more realistic and more consistent with the phoneme expression effect.

[0073] Through the above scheme, after the original lip animation is obtained by using phonemes, the weight of the original lip animation is corrected. Specifically, two correction methods are adopted, namely, first correcting the pixel effect presented by the original lip animation to obtain the reconstructed lip animation; then, using the key point shape value to correct the reconstructed lip animation again, so as to obtain the target lip animation. This scheme can effectively improve the efficiency of lip animation generation.

[0074] In one or more embodiments of the present application, based on the correspondence between the key point coordinates and the reconstructed lip animation, the key point shape value and the reconstructed lip animation weight are fused to generate a target lip animation after the original lip animation is corrected, including: according to the correspondence, by subtracting the key point shape value from the neutral shape value of the basic blend shape, determining the shape offset of the key point coordinates relative to the basic blend shape; determining a weight correction value based on the reconstructed lip animation weight and the shape offset; and using the weight correction value to correct the basic animation corresponding to the basic blend shape to generate the target lip animation.

[0075] like Figure 2 Schematic diagram of basic blending shapes for illustrating the embodiments of the present application. Figure 2 You can see that the entire face consists of many vertices, and each vertex has its corresponding basic weight value, which is the neutral shape value.

[0076] After the key point shape value is calculated, the key point shape value is subtracted from the neutral shape value in the basic blend shape. For example, for a key point A, the corresponding key point shape value is A1, and the corresponding neutral shape value in the basic blend shape is B1. The calculated shape offset AB = A1-B1, which represents the relative size relationship between the key point shape value and the neutral shape value.

[0077] Next, we need to consider the effect of the shape offset on the weight of the reconstructed lip animation. This means that the weight value needs to be adjusted or corrected based on the shape offset of each key point and the corresponding reconstructed lip animation weight.

[0078] By adding the shape offset to the weight of the reconstructed lip animation, a weight correction value can be obtained. This correction value is used to correct the base animation in the subsequent steps. Use the weight correction value to correct the base animation corresponding to the basic blend shape to generate the target lip animation: With the weight correction value, you can start to correct the base animation corresponding to the basic blend shape. This correction process may involve adjusting the keyframes, motion trajectories or parameters of the base animation to reflect the changes in the weight correction value. The corrected base animation will be closer to the target lip animation, but it may require further processing or optimization to ensure the naturalness and accuracy of the animation.

[0079] Through the above scheme, the shape offset is determined by using the key point shape value and the neutral shape value of the basic blend shape, that is, the influence of the key point shape value on the weight can be known. Then, the basic animation corresponding to the basic blend shape is corrected based on the weight correction value to obtain the target animation. It can achieve accurate correction of the key points around the mouth, and achieve accurate correction effect with a small amount of correction.

[0080] In one or more embodiments of the present application, the original lip animation weights are input into a first lip animation model, and the original lip animation weights are corrected for pixel differences using the first lip animation model to obtain reconstructed lip animation weights, including: extracting features from the original lip animation weights to obtain first original image features; inputting the first original image features into the first lip animation model to generate a predicted feature map; and performing feature correction based on the pixel differences between the predicted feature map and the real image to obtain corresponding corrected reconstructed lip animation weights.

[0081] After determining the weight of the original lip animation, feature extraction can be further performed to obtain the first original image feature. Specifically, based on the weight of the original lip animation, the function of the feature extractor is to extract its inherent features so as to better perform subsequent processing and correction. Specifically, the convolutional neural network (CNN) in deep learning or other feature extraction methods can be used to extract features from the weight of the original lip animation. Taking CNN as an example, a network structure with multiple convolutional layers can be designed to extract high-level features from the weight of the original lip animation. These features may include information such as shape, texture, and motion, which can reflect the inherent properties and laws of the lip animation. In addition to using CNN, other feature extraction methods can also be considered, such as principal component analysis (PCA) and independent component analysis (ICA). These methods can extract the main feature components from the original data for subsequent feature representation and classification tasks. After feature extraction, the first original image feature can be obtained. This feature vector can be used to input into the first lip animation model for further prediction and correction. In this way, the feature extractor can be better utilized to improve the accuracy and naturalness of the reconstructed lip animation weight.

[0082] like Figure 3a The schematic diagram of the coding model is an example of the embodiment of the present application, such as Figure 3bA schematic diagram of a decoding model for illustrating an embodiment of the present application. In one or more embodiments of the present application, after inputting the first original image feature into the first lip animation model, generating a predicted feature map includes: inputting the first original image feature into the encoding model in the first lip animation model to generate a predicted image; inputting the predicted image into the decoding model for downsampling processing to obtain the predicted feature map.

[0083] The process of inputting the first original image features into the encoding model in the first lip animation model to generate the predicted image can be further described as follows: First, the first original image features are input into an image encoder. This encoder consists of two convolutional layers, which aims to extract spatial and channel-level information from the input features. Figure 3a As shown in the figure, the encoder module is used to encode the features. The features are processed through an encoder module. This module may contain some nonlinear activation functions, such as ReLU, to enhance the robustness of the feature representation and allow the model to learn more complex feature maps. After the encoder module, the feature map is enlarged through four consecutive upsampling modules. These modules gradually increase the size of the feature map so that it gradually approaches the size of the original image. During the upsampling process, techniques such as interpolation and deconvolution can be used to increase the dimension of the pixels. After a series of processing and upsampling, the encoder outputs a predicted image. This predicted image is generated based on the potential representation learned by the encoder based on the first original image feature.

[0084] Next, Figure 3b This is a schematic diagram of a decoding model for illustrating an embodiment of the present application; this predicted image is input into a decoding model for downsampling processing to obtain a predicted feature map: Figure 3b As shown in the figure, the decoder is a process opposite to the encoder. It starts with the predicted image, gradually reduces its resolution, and extracts higher-level feature representations. The decoder contains four consecutive downsampling modules, each of which reduces the size of the feature map by half. In this process, convolution kernels are usually used for downsampling while retaining important feature information. After a series of downsampling processes, the decoder finally outputs a predicted feature map. The size of this feature map is lower than the original image, but it contains rich feature information and can be used for subsequent feature correction and weight generation.

[0085] Through the above steps, the first original image features can be converted into a predicted feature map, providing a basis for subsequent feature correction and weight generation. This encoding-decoding structure can effectively capture the features in the lip animation image (such as: spatial and channel level features) and generate a high-quality predicted feature map.

[0086] In one or more embodiments of the present application, the feature correction is performed based on the pixel difference between the feature map and the real image to obtain the corresponding corrected predicted reconstructed lip animation weight, including: inputting the feature map into a first regression network in the first lip animation model, performing difference correction on the feature map to obtain a corrected feature map; and converting the corrected feature map into the reconstructed lip animation weight having the same dimension as the original lip animation weight.

[0087] Based on the pixel difference between the feature map and the real image, feature correction is performed to obtain the corresponding corrected predicted reconstructed lip animation weights. The specific steps are as follows: First, the feature map is input into the first regression network in the first lip animation model. This regression network is a network structure specially used for pixel difference correction, which can automatically adjust and optimize according to the difference between the feature map and the real image. In the regression network, the feature map will be forward propagated through a series of neurons and weights until the output layer. In the output layer, the parameters of the network will be optimized and adjusted according to the pixel difference between the feature map and the real image to minimize the difference between the two. This process can be achieved by an optimization algorithm (such as gradient descent). After the difference correction of the regression network, the corrected feature map is obtained. This feature map already contains feature information that is closer to the real image, but there may still be some minor differences. Next, we need to convert the corrected feature map into a reconstructed lip animation weight with the same dimension as the original lip animation weight. This conversion process can be achieved by a conversion function or a mapping relationship. The specific conversion method depends on the intrinsic connection between the feature map and the weight and the data distribution.

[0088] For example, if there is a linear relationship between the feature map and the weight, linear regression or logistic regression can be used for conversion. If the relationship is nonlinear, support vector machine (SVM) or deep learning methods can be used to establish complex mapping relationships. After obtaining the weights for reconstructing the lip animation, they need to be post-processed and verified to ensure their accuracy and effectiveness. This may include steps such as weight normalization, data cleaning, and model evaluation.

[0089] Through the above steps, the weights of the reconstructed lip animation can be obtained after correction based on the pixel difference between the feature map and the real image. This method can effectively improve the accuracy and realism of the weights, making them more in line with actual needs and application scenarios.

[0090] like Figure 4A schematic diagram of a first regression network is provided for an embodiment of the present application. In one or more embodiments of the present application, the first regression network is trained based on the actual image pixel comparison result; wherein the training method of the first regression network includes: obtaining training samples, the training samples include: original lip animation weight samples, predicted image samples and key point shape value samples; inputting the training samples into the first model to be trained, and using the loss function that characterizes the pixel difference between the predicted image and the predicted image sample to train the first regression network.

[0091] When training the first regression network, we first need to prepare a set of training samples, which include original lip animation weight samples, predicted image samples and key point shape value samples. These samples are extracted from the existing historical animation dataset and are used to train and optimize the parameters of the regression network.

[0092] Then, these training samples are input into a model to be trained, usually the first lip animation model. In the model, the input samples are forward propagated through a series of neurons and weights until the output layer. Next, a loss function is defined to measure the pixel difference between the predicted image and the sample. This loss function can be the mean square error (MSE), mean absolute error (MAE), etc. Choose the appropriate loss function according to specific needs.

[0093] Then, an optimization algorithm (such as gradient descent) is used to minimize the loss function. In each iteration, the weights and parameters of the model are updated according to the gradient of the loss function to gradually reduce the difference between the predicted image and the sample. Through continuous training and optimization, the first regression network gradually learns how to adjust and correct the pixel differences according to the input feature map, thereby generating a predicted feature map that is closer to the real image. During the training process, some regularization techniques can also be used to prevent overfitting, such as weight decay, dropout, etc. At the same time, in order to improve the training efficiency and the generalization ability of the model, we can also use some data enhancement techniques to expand the training data set.

[0094] Through the above steps, the first regression network based on the actual image pixel comparison results can be trained. This method can effectively improve the accuracy and fidelity of the predicted feature map and provide a more accurate basis for subsequent weight generation.

[0095] like Figure 5A schematic diagram of a second regression network is provided for an embodiment of the present application. In one or more embodiments of the present application, the second lip animation model includes a second regression network. The inputting the original lip animation weight into the second lip animation model to obtain the key point shape value corresponding to the key point coordinates includes: based on the preset key point coordinates, the original lip animation weight is input into the second lip animation model for conversion processing to obtain the key point shape value corresponding to the key point coordinates.

[0096] In the second lip animation model, in addition to the first regression network, a second regression network is also included. This regression network is used to further correct the key point shape values ​​to improve their accuracy and realism.

[0097] Specifically, the weights of the original lip animation are input into the second lip animation model, and the model performs conversion processing according to the preset key point coordinates. This process can be achieved through specific layers or modules in the neural network, such as convolutional layers, fully connected layers, etc.

[0098] During the conversion process, the model calculates based on the input weights and outputs the key point shape values ​​corresponding to the key point coordinates. This shape value reflects the position and shape changes of the key points in the lip animation, which is crucial to the realism and naturalness of the animation.

[0099] Similar to the first regression network, the training of the second regression network also requires a set of training samples. These samples include the original lip animation weights, the true key point shape values, and the corresponding predicted key point shape values. A loss function is used to measure the difference between the predicted key point shape values ​​and the true key point shape values, such as mean square error or mean absolute error.

[0100] Then, an optimization algorithm (such as gradient descent) is used to minimize this loss function. In each iteration, the weights and parameters of the model are updated according to the gradient of the loss function to gradually reduce the difference between the predicted key point shape values ​​and the true key point shape values. Through continuous training and optimization, the second regression network gradually learns how to generate more accurate key point shape values ​​based on the input original lip animation weights.

[0101] In summary, by combining the first regression network in the first lip animation model and the second regression network in the second lip animation model, the pixel difference correction of the original lip animation weight and the optimization of the key point shape value can be achieved. This method can improve the realism and naturalness of the animation, and provide more accurate and realistic lip animation effects for animation production and game development.

[0102] Based on the same idea, the present application embodiment also provides a model training method. Figure 6A flow chart of a model training method provided in an embodiment of the present application. Figure 6 As can be seen in , the method includes the following steps:

[0103] Step 601: Obtain training samples for training a first model to be trained and a second model to be trained; wherein the training samples include: original lip animation weight samples, predicted image samples and key point shape value samples.

[0104] Step 602: The first model to be trained is trained using the training samples to obtain a first lip animation model, wherein the first lip animation model is used to modify the original lip animation weights to obtain reconstructed lip animation weights.

[0105] Step 603: Use the training samples to train the second model to be trained to obtain a second lip animation model, wherein the second lip animation model is used to determine the key point coordinates corresponding to the reconstructed lip animation weights; so that after the model training is completed, the key point coordinates and the reconstructed lip animation weights are fused to generate a target lip animation after the original lip animation is corrected.

[0106] In one or more embodiments of the present application, the first model to be trained includes a first regression network to be trained, and the second model to be trained includes a second regression network to be trained;

[0107] Outputting reconstructed lip animation weights, predicted images, and predicted key point shape values ​​when training the first model to be trained and the second model to be trained using the training samples;

[0108] The parameters of the first regression network to be trained and the second regression network to be trained are optimized using a loss function that characterizes the pixel difference between the predicted image and the predicted image sample; wherein the loss function is composed of a first sub-loss function determined based on the original lip animation weight sample and the reconstructed lip animation weight, a second sub-loss function determined based on the predicted image sample and the predicted image, and a third sub-loss function determined based on the key point shape value sample and the predicted key point shape value.

[0109] The embodiment of the present application also provides a model training method, which is used to train a first lip animation model and a second lip animation model. The specific process is as follows:

[0110] Obtain training samples: First, a set of training samples for training the first model to be trained and the second model to be trained needs to be obtained. These samples include original lip animation weight samples, predicted image samples, and key point shape value samples. These samples can be extracted from existing animation datasets and used in subsequent training and optimization processes.

[0111] Training the first lip animation model: The first model to be trained is trained using the acquired training samples to obtain the first lip animation model. The model is used to modify the original lip animation weights to obtain the reconstructed lip animation weights. During the training process, the loss function can be minimized by an optimization algorithm (such as gradient descent) to gradually reduce the pixel difference between the predicted image and the sample.

[0112] Training the second lip animation model: The second model to be trained is trained using the same training sample to obtain the second lip animation model. The model is used to determine the key point coordinates corresponding to the weights of the reconstructed lip animation. During the training process, an optimization algorithm can also be used to minimize the loss function to gradually reduce the difference between the predicted key point shape value and the sample.

[0113] Parameter optimization: When training the first model to be trained and the second model to be trained, a loss function is needed to measure the difference between the prediction result and the sample. This loss function consists of three sub-loss functions: the first sub-loss function is determined based on the original lip animation weight sample and the reconstructed lip animation weight; the second sub-loss function is determined based on the predicted image sample and the predicted image; the third sub-loss function is determined based on the key point shape value sample and the predicted key point shape value. Through the comprehensive optimization of these three sub-loss functions, the performance and realism of the model can be gradually improved.

[0114] Through the above steps, the first lip animation model and the second lip animation model can be trained. These models can effectively correct the original lip animation weights and generate more accurate and realistic reconstructed lip animation weights. This method has broad application prospects in the fields of animation production, game development, etc., and can provide more advanced technical support for the generation of lip animation.

[0115] For ease of understanding, the lip animation generation method will be described below by way of specific examples. Fig.10 The figure is a schematic diagram of the lip animation generation process for illustrating an embodiment of the present application. Fig.10 As you can see, it contains five modules:

[0116] like Fig.11 Schematic diagram of a feature extractor provided in an embodiment of the present application. The feature extractor (FeatureProjector) consists of two linear layers, which converts 49-dimensional BlendShape weights into 1024-dimensional feature vectors.

[0117] like Figure 3aAs shown in the figure, the image encoder (ImageEncoder) consists of two convolutional layers, an EncoderHead module and four upsampling modules. The input of the image encoder is the feature map after Reshape, the feature map size is (1, 32, 32), and the output is the predicted rendering image 〖Img〗_out, the predicted image size is (1, 512, 512). Figure 3b As shown in Figure 1, the image decoder (ImageDecoder) consists of two convolutional layers, a DecoderTail module and four downsampling modules. The input of the image decoder is the predicted image Img of ImageEncoder. out , the output is the downsampled feature map, and the feature map size is (1, 32, 32).

[0118] like Figure 4 As shown in Figure 2, the BlendShape regression network (BlendWeightsRegressor) consists of two linear layers, which converts the 1024-dimensional feature vector after Reshape into the reconstructed weight BS. out . Feature extractor structure.

[0119] like Figure 5 As shown, the key point regression network (LandMarkRegressor) consists of two linear layers, which converts the 49-dimensional BlendShape weights into 36-dimensional key point coordinates Ldm out .

[0120] like Figure 2 Schematic diagram of the basic blend shape provided in the embodiment of the present application. The present invention uses 49 BlendShape weights. Each BlendShape is a Mesh containing 1220 vertices and 2304 faces. A new Shape obtained by blending the BlendShape weights and BlendShape can be expressed by formula (1-1). Where BS 0 Neutral BlendShape, BS i is the vertex coordinate of the i-th BlendShape, w i is the blending weight corresponding to the i-th BlendShape. The surface structure of the new Shape is the same as that of the Neutral BlendShape.

[0121]

[0122] This application input contains BlendShape weight BS inp , the rendered image Img obtained by engine rendering and calculation inp And the mouth key point coordinates Ldminp The output is the predicted image Img out , predicted key point coordinates Ldm out , and the reconstructed weights BS out The final loss consists of the L1 loss of the image, the L2 loss of the key point coordinates, and the L1 loss of the BS, as shown in the following formula (1-2):

[0123] L=α‖Img out -Img inp ‖ 1 +β‖Ldm out -Ldm inp ‖ 2 +γ‖BS out -BS inp ‖ 1 (1-2)

[0124] Among them, α=β=γ=1 is the loss coefficient.

[0125] Based on the same idea, the present application also provides a lip animation generation device. Figure 7 A schematic diagram of the structure of a lip animation generation device provided in an embodiment of the present application. Figure 7 As can be seen, the device includes:

[0126] The acquisition module 71 is used to acquire the original lip animation weight.

[0127] The first input module 72 is used to input the original lip animation weights into a first lip animation model, and use the first lip animation model to perform pixel difference correction on the original lip animation weights to obtain reconstructed lip animation weights.

[0128] The second input module 73 is used to input the original lip animation weights into the second lip animation model to obtain key point shape values ​​corresponding to key point coordinates.

[0129] The generating module 74 is used to generate a target lip animation after the original lip animation is corrected by fusing the key point shape value and the reconstructed lip animation weight based on the correspondence between the key point coordinates and the reconstructed lip animation.

[0130] A generating module 74, configured to determine a shape offset of the key point coordinates relative to the basic blend shape by subtracting the key point shape value from the neutral shape value of the basic blend shape according to the corresponding relationship;

[0131] Determining a weight correction value based on the reconstructed lip animation weight and the shape offset;

[0132] The target lip animation is generated by modifying the basic animation corresponding to the basic blend shape using the weight correction value.

[0133] The first input module 72 is used to extract features from the original lip animation weights to obtain first original image features; after inputting the first original image features into the first lip animation model, a predicted feature map is generated; and feature correction is performed based on the pixel difference between the predicted feature map and the real image to obtain the corresponding corrected reconstructed lip animation weights.

[0134] A generating module 74, configured to input the first original image feature into a coding model in a first lip animation model to generate a predicted image;

[0135] The predicted image is input into a decoding model for downsampling processing to obtain the predicted feature map.

[0136] A first input module 72, used for inputting the feature map into a first regression network in the first lip animation model, performing difference correction on the feature map, and obtaining a corrected feature map;

[0137] The modified feature map is converted into the reconstructed lip animation weight having the same dimension as the original lip animation weight.

[0138] Optionally, a training module 75 is further included. The first regression network is trained based on the actual image pixel comparison result; wherein the training method of the first regression network includes:

[0139] Acquire training samples, wherein the training samples include: original lip animation weight samples, predicted image samples and key point shape value samples;

[0140] The training samples are input into the first model to be trained, and the first regression network is obtained by training using a loss function that characterizes the pixel difference between the predicted image and the predicted image sample.

[0141] Optionally, the second lip animation model includes a second regression network;

[0142] The second input module 73 is used to input the original lip animation weight into the second lip animation model for conversion based on the preset key point coordinates to obtain the key point shape value corresponding to the key point coordinates.

[0143] Based on the same idea, the present application embodiment also provides a model training device. Figure 8 A schematic diagram of the structure of a model training device provided in an embodiment of the present application. Figure 8 As can be seen, the device includes:

[0144] The acquisition module 81 is used to acquire training samples for training the first model to be trained and the second model to be trained; wherein the training samples include: original lip animation weight samples, predicted image samples and key point shape value samples.

[0145] The first training module 82 is used to train the first model to be trained using the training samples to obtain a first lip animation model, and the first lip animation model is used to modify the original lip animation weight to obtain a reconstructed lip animation weight.

[0146] The second training module 83 is used to train the second model to be trained using the training samples to obtain a second lip animation model, wherein the second lip animation model is used to determine the key point coordinates corresponding to the reconstructed lip animation weights; so that after the model training is completed, the key point coordinates and the reconstructed lip animation weights are fused to generate a target lip animation after the original lip animation is corrected.

[0147] Based on the above embodiment, the weight of the original lip animation is obtained; wherein the weight of the original lip animation is determined based on phonemes; the weight of the original lip animation is input into the first lip animation model, and the weight of the original lip animation is corrected for pixel differences using the first lip animation model to obtain the weight of the reconstructed lip animation; the weight of the original lip animation is input into the second lip animation model to obtain the key point shape value corresponding to the key point coordinates; based on the correspondence between the key point coordinates and the reconstructed lip animation, the key point shape value and the reconstructed lip animation weight are fused to generate the target lip animation after the original lip animation is corrected. Through the above embodiment, the weight of the original lip animation obtained by the traditional method is corrected to obtain the weight of the reconstructed lip animation; at the same time, the key point shape value corresponding to the key point coordinates needs to be used to comprehensively generate the lip animation. Since the pixel dimension compares the difference between the synthetic lip shape and the real lip shape when generating the lip animation here, the generated lip animation is more reasonable. Moreover, based on the use of weight values ​​to generate lip-sync animation, the key point regression network is further combined to impose stronger constraints on the mouth area, making the synthesis effect of the mouth closer to the real lip-sync animation.

[0148] In one possible design, the above Figure 7 The structure of the lip animation generating device shown can be implemented as an electronic device. Fig. 9 As shown, the electronic device may include: a processor 91 and a memory 92. The memory 92 stores executable code, and when the executable code is executed by the processor 91, the processor 91 can at least implement the lip animation generation method provided in the above embodiment. The structure of the electronic device may also include a communication interface 93 for communicating with other devices or communication networks.

[0149] In addition, an embodiment of the present invention provides a non-transitory machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor in a server or a computer, the processor executes the operations provided in the above embodiments. Figure 1 The corresponding lip animation generation method.

[0150] The device embodiments described above are merely illustrative, wherein the modules described as separate components may or may not be physically separated. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. Those of ordinary skill in the art may understand and implement the present invention without creative effort.

[0151] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by adding a necessary general hardware platform, and of course can also be implemented by combining hardware and software. Based on such an understanding, the above technical solution can essentially or contribute to the part in the form of a computer product, and the present invention can be implemented in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating lip animation, characterized in that: The method comprises: Obtaining an original lip animation weight; wherein the original lip animation weight is determined based on phonemes; Inputting the original lip animation weight into a first lip animation model, and using the first lip animation model to perform pixel difference correction on the original lip animation weight to obtain a reconstructed lip animation weight; Inputting the original lip animation weight into the second lip animation model to obtain key point shape values ​​corresponding to the key point coordinates; Based on the correspondence between the key point coordinates and the reconstructed lip animation, the key point shape values ​​and the reconstructed lip animation weights are fused to generate a target lip animation after the original lip animation is corrected.

2. The method according to claim 1, characterized in that Based on the correspondence between the key point coordinates and the reconstructed lip animation, the key point shape values ​​and the reconstructed lip animation weights are fused to generate a target lip animation after the original lip animation is corrected, including: According to the corresponding relationship, determining a shape offset of the key point coordinates relative to the basic blend shape by subtracting the key point shape value from the neutral shape value of the basic blend shape; Determining a weight correction value based on the reconstructed lip animation weight and the shape offset; The target lip animation is generated by modifying the basic animation corresponding to the basic blend shape using the weight correction value.

3. The method according to claim 1, characterized in that Inputting the original lip animation weight into a first lip animation model, and using the first lip animation model to perform pixel difference correction on the original lip animation weight to obtain a reconstructed lip animation weight, including: Performing feature extraction on the original lip animation weight to obtain a first original image feature; After inputting the first original image feature into the first lip animation model, generating a predicted feature map; Feature correction is performed based on the pixel difference between the predicted feature map and the real image to obtain the corresponding corrected reconstructed lip animation weight.

4. The method according to claim 3, characterized in that After inputting the first original image feature into the first lip animation model, generating a predicted feature map includes: Inputting the first original image feature into the encoding model in the first lip animation model to generate a predicted image; The predicted image is input into a decoding model for downsampling processing to obtain the predicted feature map.

5. The method according to claim 3, characterized in that: The feature correction based on the pixel difference between the feature map and the real image to obtain the corresponding corrected predicted reconstructed lip animation weight includes: Inputting the feature map into a first regression network in the first lip animation model, performing difference correction on the feature map, and obtaining a corrected feature map; The modified feature map is converted into the reconstructed lip animation weight having the same dimension as the original lip animation weight.

6. The method according to claim 5, characterized in that The first regression network is trained based on the actual image pixel comparison result; wherein the training method of the first regression network includes: Acquire training samples, wherein the training samples include: original lip animation weight samples, predicted image samples and key point shape value samples; The training samples are input into the first model to be trained, and the first regression network is obtained by training using a loss function that characterizes the pixel difference between the predicted image and the predicted image sample.

7. The method according to claim 3, characterized in that The second lip animation model includes a second regression network; The step of inputting the original lip animation weight into the second lip animation model to obtain key point shape values ​​corresponding to key point coordinates includes: Based on the preset key point coordinates, the original lip animation weights are input into the second lip animation model for conversion processing to obtain key point shape values ​​corresponding to the key point coordinates.

8. A model training method, characterized in that: The method comprises: Acquire training samples for training the first model to be trained and the second model to be trained; wherein the training samples include: original lip animation weight samples, predicted image samples and key point shape value samples; Using the training samples to train the first model to be trained to obtain a first lip animation model, wherein the first lip animation model is used to modify the original lip animation weight to obtain a reconstructed lip animation weight; The second model to be trained is trained using the training samples to obtain a second lip animation model, wherein the second lip animation model is used to determine the key point coordinates corresponding to the reconstructed lip animation weights respectively; so that after the model training is completed, the key point coordinates and the reconstructed lip animation weights are fused to generate a target lip animation after the original lip animation is corrected.

9. The method according to claim 8, characterized in that The first model to be trained includes a first regression network to be trained, and the second model to be trained includes a second regression network to be trained; Outputting reconstructed lip animation weights, predicted images, and predicted key point shape values ​​when training the first model to be trained and the second model to be trained using the training samples; The parameters of the first regression network to be trained and the second regression network to be trained are optimized using a loss function that characterizes the pixel difference between the predicted image and the predicted image sample; wherein the loss function is composed of a first sub-loss function determined based on the original lip animation weight sample and the reconstructed lip animation weight, a second sub-loss function determined based on the predicted image sample and the predicted image, and a third sub-loss function determined based on the key point shape value sample and the predicted key point shape value.

10. A lip animation generating device, characterized in that: The device comprises: The acquisition module is used to obtain the original lip animation weight; A first input module, configured to input the original lip animation weight into a first lip animation model, and use the first lip animation model to perform pixel difference correction on the original lip animation weight to obtain a reconstructed lip animation weight; A second input module, used for inputting the original lip animation weight into a second lip animation model to obtain key point shape values ​​corresponding to key point coordinates; A generation module is used to generate a target lip animation after the original lip animation is corrected by fusing the key point shape value and the reconstructed lip animation weight based on the correspondence between the key point coordinates and the reconstructed lip animation.

11. An electronic device comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 9.

12. A computer-readable medium having stored thereon at least one instruction, at least one program, code set or instruction set, wherein the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the method according to any one of claims 1 to 9.

Images