Animation display method and device based on speech-driven two-dimensional lip sync animation

By using preset 2D lip prediction model and redirection processing technology in 2D lip animation, the problem of unnatural lip transition is solved, and more natural and realistic animation effects are achieved, avoiding lip shape in traditional methods Preset limitations.

WO2024230608A1PCT designated stage expired Publication Date: 2024-11-14ZHEJIANG ZEEKR INTELLIGENT TECH CO LTD +1

Patent Information

Application Number
PCT/CN2024/090950
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-06
Filing Date
2024-04-30
Publication Date
2024-11-14

AI Technical Summary

Technical Problem

In the prior art, the lip transition in two-dimensional lip animations is unnatural, resulting in the animation being not realistic and natural enough.

Method used

By inputting voice data to preset a two-dimensional lip shape prediction model, outputting a predicted lip shape, and performing redirection processing, determining the actual lip shape of the two-dimensional character when deducing voice data, combining deep learning technology and traditional methods to achieve lip shape coherence and naturalness.

Benefits of technology

It solves the problem of unnatural character lip transition in two-dimensional lip animations, making the animation more realistic and natural, without pre-labeling the dataset or obtaining phoneme-to-view maps.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024090950_14112024_PF_FP_ABST
    Figure CN2024090950_14112024_PF_FP_ABST
Patent Text Reader

Abstract

The present application relates to the technical field of data processing. Provided are an animation display method and device based on speech-driven two-dimensional lip sync animation. The method comprises: inputting first speech data into a preset two-dimensional mouth shape prediction model, so as to output a predicted mouth shape, the first speech data being speech data of a two-dimensional character in a two-dimensional lip sync animation, and the predicted mouth shape being a mouth shape of a preset reference character in the preset two-dimensional mouth shape prediction model during representation of the first speech data; redirecting the predicted mouth shape, so as to determine an actual mouth shape of the two-dimensional character during representation of the first speech data; and during playback of the first speech data, displaying as the actual mouth shape the mouth shape of the two-dimensional character in the two-dimensional lip sync animation. The method of the present application does not need to pre-label data sets or pre-acquire phoneme-to-viseme mapping, and also achieves that output mouth shapes are independent of speakers, thus resulting in higher practicality of the preset two-dimensional mouth shape prediction model and making output mouth shapes more coherent and natural.
Need to check novelty before this filing date? Find Prior Art

Description

Animation display method and device based on voice-driven two-dimensional lip animation

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 6, 2023, with application number 202310503052X and application name “Animation display method and device based on voice-driven two-dimensional lip animation”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of data processing technology, and in particular to an animation display method and device based on voice-driven two-dimensional lip-sync animation. Background Art

[0003] The production of lip-sync animation occupies an important position in the fields of film and television animation, games, and virtual reality. Animated characters display different lip shapes according to actual voices, presenting more realistic and natural animation works to the audience.

[0004] In the prior art, the input speech is usually recognized to match the basic mouth shape and output the corresponding mouth shape animation; or the speech is converted into phonemes, and the mouth shape is determined based on the correspondence between phonemes and visemes, thereby synthesizing the lip shape animation.

[0005] The above solutions essentially determine the actual output lip shape through the mapping relationship between speech and preset lip shape. Since the output lip shape is preset, there is a problem of unnatural lip shape transition in actual application scenarios, making the lip shape animation not realistic and natural enough.

[0006] Summary of the Invention

[0007] The purpose of this application is to provide an animation display method and device based on voice-driven two-dimensional lip-sync animation, so as to solve the problem of unnatural lip transition in lip-sync animation.

[0008] In one aspect, the present application provides an animation display method based on voice-driven two-dimensional lip animation, the method comprising:

[0009] Inputting first voice data into a preset two-dimensional lip shape prediction model and outputting a predicted lip shape; wherein the first voice data is voice data of a two-dimensional character in a two-dimensional lip shape animation, and the predicted lip shape is the lip shape of a preset reference character in the preset two-dimensional lip shape prediction model when performing the first voice data;

[0010] Redirecting the predicted lip shape to determine the actual lip shape of the two-dimensional character when performing the first voice data;

[0011] When playing the first voice data, the lip shape of the two-dimensional character in the two-dimensional lip animation is displayed as the actual lip shape.

[0012] Optionally, redirecting the predicted lip shape to determine the actual lip shape of the two-dimensional character when performing the first voice data includes:

[0013] Determining a key point coordinate displacement vector of the predicted lip shape relative to the preset reference lip shape based on the key point coordinate vector of the predicted lip shape and the key point coordinate vector of the preset reference lip shape of the preset reference character;

[0014] determining actual coordinate vectors of the key points of the lip shape of the two-dimensional character when performing the first voice data based on the key point coordinate displacement vector and the key point coordinate vector of the preset reference lip shape of the two-dimensional character;

[0015] The lip shape indicated by the actual coordinate vector of the key point is determined as the actual lip shape.

[0016] Optionally, redirecting the predicted lip shape to determine the actual lip shape of the two-dimensional character when performing the first voice data includes:

[0017] Determine, based on the first preset key point basic coordinate vector, the second preset key point basic coordinate vector, the key point coordinate vector of the predicted lip shape, and the preset weight vector, the actual coordinate vector of the key points of the lip shape of the two-dimensional character when interpreting the first voice data; wherein the first preset key point basic coordinate vector is a vector of the key point coordinates of n predetermined basic lip shapes of the two-dimensional character based on the preset reference lip shape and the preset standard deviation, where n is a non-zero natural number; the second preset key point basic coordinate vector is a vector of the key point coordinates of n predetermined basic lip shapes of the preset reference character based on the preset reference lip shape and the preset standard deviation; and the preset weight vector is a vector of the weights of the basic lip shapes of the two-dimensional character;

[0018] The lip shape indicated by the actual coordinate vector of the key point is determined as the actual lip shape.

[0019] Optionally, the method further includes:

[0020] Obtaining a three-dimensional audiovisual dataset and preprocessing the three-dimensional audiovisual dataset to obtain training data for the preset two-dimensional lip prediction model; wherein the three-dimensional audiovisual dataset includes three-dimensional video data and second voice data, the second voice data being voice data of a three-dimensional character in the three-dimensional video data; and the training data includes input data and calibration data;

[0021] An initial model of the preset two-dimensional lip shape prediction model is trained according to the training data until a preset condition is met, and the preset two-dimensional lip shape prediction model is output.

[0022] Optionally, preprocessing the three-dimensional audio-visual data set to obtain training data for the preset two-dimensional lip shape prediction model includes:

[0023] Extracting a character image of a three-dimensional character corresponding to the second voice data from the three-dimensional video data, and intercepting a partial character image from the character image; wherein the partial character image includes the lower half of the face below the nose of the three-dimensional character;

[0024] Obtaining coordinates of initial key points of the lower half of the face in the partial character image, performing alignment processing on the lower half of the face in the partial character image, obtaining an aligned partial character image, and determining true key point coordinate vectors of the aligned partial character image;

[0025] The second speech data is determined as the input data, and the real key point coordinate vector is determined as the calibration data.

[0026] Optionally, performing alignment processing on the lower half of the face in the partial character image includes:

[0027] Determining the center point and centroid of the lower half of the face according to the coordinates of the initial key points;

[0028] According to the center point and center of mass of the lower half of the face and the center point and center of mass of the preset average human face, the coordinates of the initial key points are repeatedly adjusted until the lower half of the face is aligned with the preset average human face.

[0029] Optionally, training an initial model of the preset two-dimensional lip shape prediction model according to the training data until a preset condition is met, and outputting the preset two-dimensional lip shape prediction model, comprises:

[0030] Inputting the second speech data into the initial model of the preset two-dimensional lip shape prediction model, and outputting a predicted key point coordinate vector; wherein the predicted key point coordinate vector is the key point coordinate vector of the lip shape of the preset reference character when performing the second speech data;

[0031] Substitute the predicted key point coordinate vector and the real key point coordinate vector into a preset loss function to determine a loss value;

[0032] According to the loss value, the model parameters of the initial model are adjusted, and the model when the loss value is determined to be the preset two-dimensional lip shape prediction model is determined to be the preset two-dimensional lip shape prediction model.

[0033] On the other hand, the present application provides an animation display device based on voice-driven two-dimensional lip animation, the device comprising:

[0034] a prediction unit, configured to input first speech data into a preset two-dimensional lip shape prediction model and output a predicted lip shape; wherein the first speech data is speech data of a two-dimensional character in a two-dimensional lip shape animation, and the predicted lip shape is the lip shape of a preset reference character in the preset two-dimensional lip shape prediction model when performing the first speech data;

[0035] a redirection unit, configured to redirect the predicted lip shape to determine the actual lip shape of the two-dimensional character when performing the first voice data;

[0036] The display unit is configured to display the lip shape of the two-dimensional character in the two-dimensional lip animation as the actual lip shape when playing the first voice data.

[0037] Optionally, the redirection unit includes a first redirection module, and the first redirection module includes a first determination module, a second determination module, and a third determination module;

[0038] The first determining module is configured to determine a key point coordinate displacement vector of the predicted lip shape relative to the preset reference lip shape based on the key point coordinate vector of the predicted lip shape and the key point coordinate vector of the preset reference lip shape of the preset reference character;

[0039] The second determining module is configured to determine the actual coordinate vectors of the key points of the lip shape of the two-dimensional character when performing the first voice data based on the key point coordinate displacement vector and the key point coordinate vector of the preset reference lip shape of the two-dimensional character;

[0040] The third determining module is used to determine that the lip shape indicated by the actual coordinate vector of the key point is the actual lip shape.

[0041] Optionally, the redirection unit includes a second redirection module, and the second redirection module includes a first processing module and a second processing module;

[0042] The first processing module is configured to determine, based on a first preset key point basic coordinate vector, a second preset key point basic coordinate vector, the key point coordinate vector of the predicted mouth shape, and a preset weight vector, the actual coordinate vector of the key points of the mouth shape of the two-dimensional character when interpreting the first voice data; wherein the first preset key point basic coordinate vector is a vector of key point coordinates of n predetermined basic mouth shapes of the two-dimensional character based on a preset reference mouth shape and a preset standard deviation, where n is a non-zero natural number; the second preset key point basic coordinate vector is a vector of key point coordinates of n predetermined basic mouth shapes of preset reference characters based on a preset reference mouth shape and a preset standard deviation; and the preset weight vector is a vector of weights of the basic mouth shapes of the two-dimensional character;

[0043] The second processing module is used to determine that the mouth shape indicated by the actual coordinate vector of the key point is the actual mouth shape.

[0044] Optionally, the device further comprises a model training unit, wherein the model training unit comprises an acquisition module and a training module;

[0045] The acquisition module is configured to acquire a three-dimensional audio-visual dataset and pre-process the three-dimensional audio-visual dataset to obtain training data for the preset two-dimensional lip prediction model; wherein the three-dimensional audio-visual dataset includes three-dimensional video data and second voice data, wherein the second voice data is voice data of a three-dimensional character in the three-dimensional video data; and the training data includes input data and calibration data;

[0046] The training module is used to train the initial model of the preset two-dimensional lip shape prediction model according to the training data, and output the preset two-dimensional lip shape prediction model when a preset condition is met.

[0047] Optionally, the acquisition module includes an extraction module, an alignment module, and a determination module;

[0048] The extraction module is configured to extract a character image of a three-dimensional character corresponding to the second voice data from the three-dimensional video data, and intercept a partial character image from the character image; wherein the partial character image includes the lower half of the face below the nose of the three-dimensional character;

[0049] The alignment module is used to obtain the coordinates of the initial key points of the lower half of the face in the partial character image, perform alignment processing on the lower half of the face in the partial character image, obtain the aligned partial character image, and determine the real key point coordinate vectors of the aligned partial character image;

[0050] The determination module is used to determine the second speech data as the input data, and determine the real key point coordinate vector as the calibration data.

[0051] Optionally, the alignment module is specifically used to determine the center point and center of mass of the lower half of the face based on the coordinates of the initial key points; and repeatedly adjust the coordinates of the initial key points based on the center point and center of mass of the lower half of the face and the center point and center of mass of a preset average face until the lower half of the face is aligned with the preset average face.

[0052] Optionally, the training module includes an input module, a calculation module, and an output module;

[0053] The input module is configured to input the second speech data into the initial model of the preset two-dimensional lip shape prediction model and output a predicted key point coordinate vector; wherein the predicted key point coordinate vector is a key point coordinate vector of the lip shape of the preset reference character when performing the second speech data;

[0054] The calculation module is used to bring the predicted key point coordinate vector and the real key point coordinate vector into a preset loss function to determine the loss value;

[0055] The output module is used to adjust the model parameters of the initial model according to the loss value, and determine that the model when the loss value is the target value is the preset two-dimensional lip shape prediction model.

[0056] On the other hand, the present application further provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0057] The memory stores computer-executable instructions;

[0058] The processor executes the computer-executable instructions stored in the memory to implement the method described above.

[0059] On the other hand, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to implement the above-mentioned method when executed by a processor.

[0060] On the other hand, the present application also provides a computer program product, including a computer program, which is used to implement the above method when executed by a processor.

[0061] On the other hand, the present application also provides a computer program, which implements any of the above methods when executed by a processor.

[0062] The present application provides an animation display method and device for voice-driven two-dimensional lip-sync animation, comprising: inputting first voice data into a preset two-dimensional lip-sync prediction model and outputting predicted lip shapes; wherein the first voice data is the voice data of a two-dimensional character in the two-dimensional lip-sync animation, and the predicted lip shapes are the lip shapes of a preset reference character in the preset two-dimensional lip-sync prediction model when performing the first voice data; redirecting the predicted lip shapes to determine the actual lip shapes of the two-dimensional character when performing the first voice data; and when playing the first voice data, displaying the lip shapes of the two-dimensional character in the two-dimensional lip-sync animation as the actual lip shapes. The solution of the present application predicts lip shapes by using a preset two-dimensional lip-sync prediction model, combining the advantages of traditional methods and deep learning methods. It not only eliminates the need for pre-labeling of datasets and pre-acquiring phoneme-to-viseme mappings, but also achieves speaker-independent lip shape output, making the preset two-dimensional lip-sync prediction model more practical. Moreover, the output lip shapes are more coherent and natural, solving the problem of unnatural lip shape transitions in 2D lip-sync animations in the prior art, and making the displayed 2D lip-sync animations more realistic. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0064] FIG1 is a flow chart of an animation display method based on voice-driven two-dimensional lip-sync animation provided in an embodiment of the present application;

[0065] FIG2 is a schematic diagram of a network architecture of a preset two-dimensional lip shape prediction model provided in an embodiment of the present application;

[0066] FIG3 is a flow chart of a method for training a model provided in an embodiment of the present application;

[0067] FIG4 is a schematic diagram of a preset reference lip shape provided in an embodiment of the present application;

[0068] FIG5 is a schematic structural diagram of an animation display device based on voice-driven two-dimensional lip-sync animation provided by an embodiment of the present application;

[0069] FIG6 is a schematic structural diagram of another animation display device based on voice-driven two-dimensional lip-sync animation provided in an embodiment of the present application;

[0070] FIG7 is a schematic structural diagram of an electronic device provided in an embodiment of the present application.

[0071] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0072] Voice-driven 2D lip animation is currently a major form of 2D lip-syncing, widely used in film and television animation, games, virtual reality, and other fields. Voice-driven 2D lip animation generates 2D animated lip shapes based on a voice input. "2D" stands for "2 Dimensions," also known as two-dimensional.

[0073] In one example, lip sync technology is used to synthesize lip animation. There are many different lip sync implementations, such as Rhubarb Lip Sync and uLipSync. For example, Rhubarb Lip Sync is based on the six lip shapes invented by Hanna-Barbera. It recognizes input speech and matches it with the six lip shapes invented by Hanna-Barbera, as well as three extended lip shapes, to output a fixed-format 2D lip animation file.

[0074] In another example, 22 visemes are provided, each describing the mouth position of a specific set of phonemes. Each viseme corresponds to an ID, and the viseme ID is mapped to the corresponding International Phonetic Alphabet phoneme through Speech Synthesis Markup Language (SSML). The viseme ID and offset timestamp can be attached to the speech audio output, so that the corresponding scalable lip shape vector graphics can be obtained through the viseme ID, and then the time-based facial features can be obtained to synthesize the lip shape animation.

[0075] In another example, deep learning techniques are used to learn audio data and 2D lip animations, determining the mapping between phonemes and lip shapes. For example, a three-stage Long Short-Term Memory Network (LSTM) is used to map the input speech data to a 2D JALI-based viseme representation. Subsequently, when synthesizing the lip animation, the speech is recognized and the actual output lip shape is determined based on the mapping between phonemes and lip shapes.

[0076] The above examples all achieve lip synchronization through the mapping relationship between phonemes and preset lip-sync visemes, thereby synthesizing 2D lip-sync animation. Since the actual output lip shapes are all preset, due to the limited number of preset lip-sync visemes, in actual application scenarios, there is a problem of unnatural lip transitions, making the generated lip-sync animation less realistic and natural.

[0077] In order to solve the above problems, the present application provides an animation display method based on voice-driven two-dimensional lip-sync animation, which predicts the lip shape of the voice data of a 2D character through a pre-trained 2D lip-sync prediction model, and redirects the predicted lip shape with the 2D character so that the 2D character displays the lip shape predicted by the model when performing the voice data. Since the lip shapes of the 2D characters are all predicted by the 2D lip-sync prediction model, they are inherently coherent and there will be no unnatural lip shape transitions. This solves the problem of unnatural character lip shape transitions in the 2D lip-sync animation of the prior art and makes the displayed 2D lip-sync animation more realistic.

[0078] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0079] Figure 1 is a flow chart of an animation display method based on voice-driven two-dimensional lip-syncing animation provided in an embodiment of the present application. The execution subject of the method provided in the embodiment of the present application can be an animation display device based on voice-driven two-dimensional lip-syncing animation, and the device can be located on an electronic device, which can be a terminal device such as a computer, a mobile phone, or a tablet, and this application does not limit it. The embodiment of the present application takes the execution subject as an animation display device based on voice-driven two-dimensional lip-syncing animation (hereinafter referred to as the display device) as an example for detailed introduction.

[0080] As shown in FIG1 , the animation display method based on voice-driven two-dimensional lip animation provided in this embodiment includes:

[0081] S101. Input first voice data into a preset two-dimensional lip shape prediction model and output a predicted lip shape; wherein the first voice data is voice data of a two-dimensional character in a two-dimensional lip shape animation, and the predicted lip shape is the lip shape of a preset reference character in the preset two-dimensional lip shape prediction model when performing the first voice data.

[0082] Exemplarily, to determine the actual lip shape of a two-dimensional character when performing speech, the display device first inputs the speech data of the two-dimensional character in the two-dimensional lip animation, that is, the first speech data, into a preset two-dimensional lip shape prediction model. The preset two-dimensional lip shape prediction model predicts the lip shape of the preset reference character when performing the first speech data based on the first speech data, and outputs the predicted lip shape.

[0083] Among them, the preset two-dimensional lip shape prediction model is a pre-trained deep learning model used to predict lip shape based on voice data. It can adopt the Transformer network architecture or other suitable structures, and this application does not impose any restrictions.

[0084] For example, FIG2 is a schematic diagram of the network architecture of a preset two-dimensional lip shape prediction model provided in an embodiment of the present application. As shown in FIG2, the first speech data can be preprocessed by a temporal convolutional neural network (TCN), and the preprocessed speech data is input into the Transformer model. After sequentially passing through the linear interpolation layer (linear interpolation), the multi-head self-attention mechanism layer (MH self-attention), the feed forward layer (feed forward), and the linear interpolation layer (linear interpolation) in the Transformer model, the speech vector x required for model prediction can be obtained, where x = {a1, a2, ..., a T}, and then input the speech vector x into the biased cross-modal multi-head attention layer. At the same time, the two-dimensional lip shape representation after encoding before the current frame is The input is sent to the motion encoder of the Transformer model, and then processed by the periodic positional encoding layer, the biased causal MH self-attention layer, and then the biased cross-modal multi-head attention layer. The input speech vector x is then processed by the feed forward layer and the motion decoder, and the final predicted lip sequence is output in sequence. The output lip shape is the lip shape of the reference character preset in the model when interpreting the input voice data.

[0085] By using the model to predict the actual lip shape of speech data, the consistency and naturalness of the lip shape can be improved, making the lip shape animation more natural and realistic.

[0086] S102: Redirect the predicted lip shape to determine the actual lip shape of the two-dimensional character when performing the first voice data.

[0087] For example, since the output predicted lip shape is the lip shape of the preset reference character in the preset two-dimensional lip shape prediction model when performing the first voice data, and the two-dimensional character in the two-dimensional lip shape animation is usually different from the preset reference character, the predicted lip shape needs to be redirected to determine the actual lip shape of the two-dimensional character when performing the first voice data.

[0088] The present application does not limit how to redirect the predicted lip shape. In one example, redirecting the predicted lip shape to determine the actual lip shape of the two-dimensional character when performing the first voice data may include:

[0089] S1. Determine a key point coordinate displacement vector of the predicted lip shape relative to the preset reference lip shape based on a key point coordinate vector of the predicted lip shape and a key point coordinate vector of a preset reference lip shape of a preset reference character.

[0090] S2. Determine the actual coordinate vectors of the key points of the lip shape of the two-dimensional character when interpreting the first voice data based on the key point coordinate displacement vector and the key point coordinate vector of the preset reference lip shape of the two-dimensional character.

[0091] S3. Determine that the lip shape indicated by the actual coordinate vector of the key point is the actual lip shape.

[0092] For example, the key points of the preset reference lip shape of the reference average face shape model (i.e., the preset reference character) used in training can be bound to the key points of the preset reference lip shape r0 of the target average face shape model (i.e., the two-dimensional character). The preset reference lip shape can be the lip shape of the two-dimensional character when not speaking or the lip shape of the two-dimensional character in a natural state. Then, the predicted lip shape output by the preset two-dimensional lip shape prediction model is calculated for each frame. The displacement d of the key points relative to the preset reference lip shape s0 of the preset reference character, that is, the key point coordinate displacement vector can be obtained by subtracting the key point coordinate vector of the preset reference lip shape from the key point coordinate vector of the predicted lip shape. Among them, the preset reference lip shape can be the lip shape of the preset reference character when not speaking or the lip shape of the preset reference character in a natural state. Finally, based on the preset reference lip shape r0 and the displacement d of the two-dimensional character, the coordinate vector r of the key points of the actual lip shape of the two-dimensional character corresponding to each frame can be obtained. i = r0 + d, where i = 1, 2, ..., T. That is, the key point coordinate vectors of the preset reference lip shape of the two-dimensional character are added to the key point coordinate displacement vectors to obtain the actual key point coordinate vectors of the lip shape of the two-dimensional character when interpreting the first voice data, thereby determining the actual lip shape of the two-dimensional character when interpreting the first voice data.

[0093] In another example, redirecting the predicted lip shape to determine the actual lip shape of the two-dimensional character when performing the first voice data may include:

[0094] S10. Determine the actual coordinate vector of the key points of the lip shape of the two-dimensional character when interpreting the first voice data based on the first preset key point basic coordinate vector, the second preset key point basic coordinate vector, the key point coordinate vector of the predicted lip shape, and the preset weight vector; wherein the first preset key point basic coordinate vector is a vector of the key point coordinates of the n predetermined basic lip shapes of the two-dimensional character based on the preset reference lip shape and the preset standard deviation, where n is a non-zero natural number; the second preset key point basic coordinate vector is a vector of the key point coordinates of the n predetermined basic lip shapes of the preset reference character based on the preset reference lip shape and the preset standard deviation; the preset weight vector is a vector of the weights of the basic lip shapes of the two-dimensional character.

[0095] S20: Determine that the lip shape indicated by the actual coordinate vector of the key point is the actual lip shape.

[0096] For example, rig-space retargeting can be used during retargeting. Rig-space retargeting requires defining several basic reference shape models. In this application, n corresponding basic lip shapes are predefined for the preset reference character and the two-dimensional character, respectively.

[0097] For example, 8 basic lip shapes with standard deviations of +3 and -3 relative to the preset reference lip shape s0 can be defined for the preset reference character. in, Then any basic lip shape of the preset reference character can be expressed as Among them, s0 is the coordinate vector of the key points of the preset reference mouth shape, and each s i is the shape basis vector, p i is the shape parameter, and m is the number of shapes. That is, the first preset key point basic coordinate vector is

[0098] Similarly, 8 basic lip shapes with standard deviations of +3 and -3 relative to the preset reference lip shape r0 are defined for the 2D character. That is, the second preset key point basic coordinate vector is

[0099] After defining the corresponding basic lip shapes for the preset reference character and the two-dimensional character respectively, the predicted lip shape of the preset reference character predicted by the preset two-dimensional lip shape prediction model of this application can be directly mapped to the lip shape of the two-dimensional character through a linear combination of R, such as rig-space interpolation.

[0100] Among them, the weight of the basic lip shape of each two-dimensional character The calculation formula is as follows:

[0101] In this formula, It is the key point coordinate vector of the predicted lip shape predicted by the preset two-dimensional lip shape prediction model of this application, u∈{-3,+3}, the value of u depends on whether the shape is a positive standard deviation or a negative standard deviation relative to the preset reference lip shape, when it is a positive standard deviation, u is +3, when it is a negative standard deviation, u is -3.

[0102] Finally, the actual coordinate vector R of the key points of the lip shape of the two-dimensional character when interpreting the first voice data t The calculation formula is: t =(R-r0)w+r0

[0103] In this formula, w is a vector of weights of the basic lip shapes of the above eight two-dimensional characters based on rig-space interpolation, that is, a preset weight vector in this application.

[0104] Based on this, the above formula ultimately yields the actual coordinate vectors of the key points of the 2D character's lip shape when interpreting the first speech data, thereby determining the 2D character's actual lip shape when interpreting the first speech data. Using animation retargeting technology, the lip shape animation can be transferred to different 2D characters, eliminating the need for extensive manual lip shape presetting. This ensures speaker-independent lip shape output, making the pre-set 2D lip shape prediction model more practical and applicable to 2D lip shape animations of various shape and appearance models.

[0105] S103: When playing the first voice data, display the lip shape of the two-dimensional character in the two-dimensional lip animation as the actual lip shape.

[0106] For example, after determining the actual lip shape of the two-dimensional character when performing the first voice data, when playing the first voice data, the lip shape of the two-dimensional character in the two-dimensional lip animation is displayed as the actual lip shape, thus realizing voice-driven two-dimensional lip animation.

[0107] The embodiment of the present application provides an animation display method for voice-driven two-dimensional lip-sync animation, comprising: inputting first voice data into a preset two-dimensional lip-sync prediction model and outputting predicted lip-sync; wherein the first voice data is the voice data of a two-dimensional character in the two-dimensional lip-sync animation, and the predicted lip-sync is the lip-sync of a preset reference character in the preset two-dimensional lip-sync prediction model when performing the first voice data; redirecting the predicted lip-sync to determine the actual lip-sync of the two-dimensional character when performing the first voice data; and when playing the first voice data, displaying the lip-sync of the two-dimensional character in the two-dimensional lip-sync animation as the actual lip-sync. The solution of the present application predicts lip-sync by using a preset two-dimensional lip-sync prediction model, combining the advantages of traditional methods and deep learning methods. It not only does not require pre-labeling of data sets, but also does not require pre-acquisition of phoneme-to-viseme mappings, and also achieves that the output of lip-sync is independent of the speaker, making the preset two-dimensional lip-sync prediction model more practical. Moreover, the output lip-sync is more coherent and natural, solving the problem of unnatural lip-sync transitions in 2D lip-sync animation in the prior art, and making the displayed 2D lip-sync animation more realistic.

[0108] Based on the above embodiment, before using the preset two-dimensional lip shape prediction model to predict the lip shape of speech data, it is necessary to train the initial model of the preset two-dimensional lip shape prediction model to obtain a model with higher prediction accuracy.

[0109] For example, FIG3 is a flow chart of a method for training a model provided in an embodiment of the present application. As shown in FIG3 , the process of training an initial model of a preset two-dimensional lip shape prediction model may include:

[0110] S301. Obtain a three-dimensional audio-visual dataset and pre-process the dataset to obtain training data for a preset two-dimensional lip prediction model. The three-dimensional audio-visual dataset includes three-dimensional video data and second voice data, where the second voice data is voice data of a three-dimensional character in the three-dimensional video data. The training data includes input data and calibration data.

[0111] Exemplarily, before model training, training data must be acquired. When acquiring training data, a three-dimensional audiovisual dataset, also referred to as a 3D audiovisual dataset, is first acquired. The 3D audiovisual dataset includes three-dimensional video data and secondary voice data, where the secondary voice data is the voice data of a three-dimensional character in the three-dimensional video data. The 3D audiovisual dataset is then preprocessed to obtain training data for an initial model used to train a pre-set two-dimensional lip-sync prediction model. The training data includes at least input data and calibration data.

[0112] The present application does not limit how to pre-process the 3D audiovisual dataset. In one example, pre-processing the 3D audiovisual dataset to obtain training data for a preset 2D lip prediction model may include:

[0113] S100, extracting a character image of a three-dimensional character corresponding to the second voice data from the three-dimensional video data, and capturing a partial character image from the character image; wherein the partial character image includes the lower half of the face below the nose of the three-dimensional character.

[0114] S200, obtaining the coordinates of the initial key points of the lower half of the face in the partial character image, aligning the lower half of the face in the partial character image, obtaining the aligned partial character image, and determining the real key point coordinate vector of the aligned partial character image.

[0115] S300: Determine the second speech data as input data, and determine the real key point coordinate vector as calibration data.

[0116] For example, the present application predicts lip shape based on voice data, which only involves the lower half of the face, that is, the face below the nose. However, the character images in the three-dimensional video data mostly include images of the entire character. Therefore, each frame of the character image of the three-dimensional character corresponding to the second voice data in the three-dimensional video data can be extracted first, and the partial character image including the lower half of the face below the nose of the three-dimensional character in each frame of the character image can be intercepted.

[0117] Then, to eliminate the effects of position, size, and orientation on the detected 3D character's face, the lower half of the captured partial character image can be aligned. Prior to alignment, key point detection is performed on the lower half of the captured partial character image to obtain the coordinates of the initial key points of the lower half of the partial character image. Alignment is then performed on the lower half of the face to determine the true key point coordinate vectors of the aligned partial character image.

[0118] Among them, key point detection can use tools such as opencv and dlib, and this application does not impose any restrictions. Alignment processing can be implemented using algorithms such as Active Appearance Model (AAM), Active Shape Model (ASM), Deep Convolutional Neural Network (Deep CNN), Tasks-Constrained Deep Convolutional Network (TCDCN), Multi-task Convolutional Neural Network (MTCNN), and Tweaked Convolutional Neural Networks (TCNN), and this application does not impose any restrictions.

[0119] In one example, aligning the lower half of a face in a partial character image may include:

[0120] S210: Determine the center point and centroid of the lower half of the face based on the coordinates of the initial key points.

[0121] S220 , repeatedly adjusting the coordinates of the initial key points according to the center point and centroid of the lower half of the face and the center point and centroid of the preset average face until the lower half of the face is aligned with the preset average face.

[0122] For example, any one of the lower halves of the face in all the captured partial character images (generally the shape in the natural state) can be selected as the preset average face shape; the preset average face can also be recalculated based on the aligned face shape, which is not limited in this application.

[0123] During alignment, you can first calculate the center point and center of mass of the lower half of the face in each frame based on the coordinates of the initial key points of the lower half of the face in each frame of the local character image, and then repeatedly adjust the coordinates of the initial key points based on the center point and center of mass of the lower half of the face and the center point and center of mass of the preset average face until convergence, or the number of adjustments reaches a limit, or the convergence time stops. At this time, it is considered that the lower half of the face is aligned with the preset average face.

[0124] Adjusting the coordinates of the initial key points includes translating, rotating, and scaling the lower half of the face to align it with a preset average face. Specifically, the center points of the key points of the lower half of the face in each frame can be aligned with the preset average face, the center of mass of the lower half of the face in each frame can be normalized, and the lower half of the face in each frame can be rotated. The adjustments can be repeated multiple times until alignment is achieved. The normalization of the facial shape generally uses Generalized Procrustes analysis (GPA), which appropriately translates, rotates, and scales the key points of the facial shape to change the non-shape interference of the original shape without changing the point distribution model.

[0125] For example, after obtaining the aligned partial character image, tools such as OpenCV and DLib are used to determine the true keypoint coordinate vectors of the aligned partial character image. After determining the true keypoint coordinate vectors of the aligned partial character image, calibration data is used as training data, and the second speech data is used as input data to obtain training data. Training the model using the true keypoint coordinate vectors of the aligned partial character image can improve the model's prediction accuracy.

[0126] Furthermore, after obtaining the lower half of the face in the aligned partial character image, principal component analysis (PCA) can be performed on each frame of the lower half face to remove redundancy and noise. Furthermore, an average shape model can be determined based on the lower half of the face in the aligned partial character image to serve as a preset reference lip shape for a preset reference character in the preset two-dimensional lip shape prediction model.

[0127] For example, FIG4 is a schematic diagram of a preset reference lip shape provided by an embodiment of the present application. As shown in FIG4, a plane coordinate system (x, y) can be set in the figure, wherein the x-axis is the horizontal axis and the y-axis is the vertical axis. The key point coordinate vector of the preset reference lip shape of the preset reference character can be expressed as: s = {x1, y1, x2, y2, ..., x N ,y N} T , where N is the number of key points, x and y are the coordinates of the key points in the coordinate system, and T is the sequence frame number.

[0128] S302: Train an initial model of a preset two-dimensional lip shape prediction model according to the training data until a preset condition is met, and then output the preset two-dimensional lip shape prediction model.

[0129] For example, after the training data is determined, the training data can be used to train the initial model of the preset two-dimensional lip shape prediction model. When the preset conditions are met, that is, the gap between the training value output by the model and the true value is small enough, the model at this time is determined to be the output preset two-dimensional lip shape prediction model for subsequent lip shape prediction.

[0130] In one example, step S302, training an initial model of a preset two-dimensional lip shape prediction model according to training data until a preset condition is met and outputting the preset two-dimensional lip shape prediction model, may include:

[0131] S3021. Input the second voice data into the initial model of a preset two-dimensional lip shape prediction model, and output a predicted key point coordinate vector; wherein the predicted key point coordinate vector is the key point coordinate vector of the lip shape of the preset reference character when interpreting the second voice data.

[0132] S3022: Substitute the predicted key point coordinate vector and the true key point coordinate vector into a preset loss function to determine the loss value.

[0133] S3023. Adjust the model parameters of the initial model according to the loss value, and determine that the model when the loss value is the target value is the preset two-dimensional lip shape prediction model.

[0134] For example, the second voice data is input into the initial model of the preset two-dimensional lip shape prediction model, and the model processes it and outputs the key point coordinate vector of the lip shape of the preset reference character when interpreting the second voice data. Then, the key point coordinate vector of the predicted mouth shape is and the true keypoint coordinate vector Y T =(y1,……,y t ) into the preset loss function to calculate the loss value.

[0135] Among them, the formula of the preset loss function is as follows:

[0136] In this formula, V is the number of key points of the lip shape of the preset reference character, T is the sequence frame number, and t and v are the sequence numbers of the key points.

[0137] After calculating the loss value, the model parameters of the initial model are adjusted accordingly until the loss value reaches the target value, ending the training. A mapping function h(x) exists between the model's input and output, and the purpose of model training is to determine the parameters of this mapping function h(x). When the gap between the predicted value and the true value is sufficiently small, the parameters of the mapping function h(x) at this point are determined as the final model parameters, and the model at this point is output as the preset two-dimensional lip shape prediction model.

[0138] In the present application, a three-dimensional audio-visual data set is processed to obtain training data for a preset two-dimensional lip shape prediction model, thereby training an initial model of the preset two-dimensional lip shape prediction model to obtain a final preset two-dimensional lip shape prediction model, so as to facilitate subsequent lip shape prediction without the need for a large number of manually preset lip shapes, thereby achieving the output of the lip shape being independent of the speaker and making the preset two-dimensional lip shape prediction model more practical; moreover, through model prediction, the output lip shape is made more coherent and natural, solving the problem of unnatural lip shape transition of characters in 2D lip shape animation in the prior art, and making the displayed 2D lip shape animation more realistic.

[0139] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0140] FIG5 is a schematic diagram of the structure of an animation display device based on voice-driven 2D lip-sync animation provided in an embodiment of the present application. As shown in FIG5 , the animation display device 50 based on voice-driven 2D lip-sync animation provided in this embodiment includes: a prediction unit 501, a redirection unit 502, and a display unit 503.

[0141] Among them, the prediction unit 501 is used to input the first voice data into a preset two-dimensional lip shape prediction model and output the predicted lip shape; wherein the first voice data is the voice data of a two-dimensional character in a two-dimensional lip shape animation, and the predicted lip shape is the lip shape of a preset reference character in the preset two-dimensional lip shape prediction model when performing the first voice data.

[0142] The redirection unit 502 is configured to perform redirection processing on the predicted lip shape to determine the actual lip shape of the two-dimensional character when performing the first voice data.

[0143] The display unit 503 is configured to display the lip shapes of the two-dimensional character in the two-dimensional lip animation as actual lip shapes when playing the first voice data.

[0144] The device provided in this embodiment can be used to execute the method of the above embodiment. Its implementation principle and technical effects are similar and will not be described in detail here.

[0145] FIG6 is a schematic diagram of the structure of another animation display device based on voice-driven 2D lip-sync animation provided in an embodiment of the present application. As shown in FIG6 , the animation display device 60 based on voice-driven 2D lip-sync animation provided in this embodiment includes: a prediction unit 601, a redirection unit 602, and a display unit 603.

[0146] Among them, the prediction unit 601 is used to input the first voice data into a preset two-dimensional lip shape prediction model and output the predicted lip shape; wherein the first voice data is the voice data of a two-dimensional character in a two-dimensional lip shape animation, and the predicted lip shape is the lip shape of a preset reference character in the preset two-dimensional lip shape prediction model when performing the first voice data.

[0147] The redirection unit 602 is configured to perform redirection processing on the predicted lip shape to determine the actual lip shape of the two-dimensional character when performing the first voice data.

[0148] The display unit 603 is configured to display the lip shapes of the two-dimensional character in the two-dimensional lip animation as actual lip shapes when playing the first voice data.

[0149] In an example, the redirection unit 602 includes a first redirection module 6021 , and the first redirection module 6021 includes a first determination module 60211 , a second determination module 60212 , and a third determination module 60213 .

[0150] The first determination module 60211 is used to determine the key point coordinate displacement vector of the predicted lip shape relative to the preset reference lip shape based on the key point coordinate vector of the predicted lip shape and the key point coordinate vector of the preset reference lip shape of the preset reference character.

[0151] The second determining module 60212 is used to determine the actual coordinate vectors of the key points of the lip shape of the two-dimensional character when interpreting the first voice data based on the key point coordinate displacement vector and the key point coordinate vector of the preset reference lip shape of the two-dimensional character.

[0152] The third determining module 60213 is used to determine that the lip shape indicated by the actual coordinate vector of the key point is the actual lip shape.

[0153] In an example, the redirection unit 602 includes a second redirection module 6022 , and the second redirection module 6022 includes a first processing module 60221 and a second processing module 60222 .

[0154] The first processing module 60221 is used to determine the actual coordinate vector of the key points of the mouth shape of the two-dimensional character when interpreting the first voice data based on the first preset key point basic coordinate vector, the second preset key point basic coordinate vector, the key point coordinate vector of the predicted mouth shape, and the preset weight vector; wherein the first preset key point basic coordinate vector is a vector of the key point coordinates of the n predetermined basic mouth shapes of the two-dimensional character based on the preset reference mouth shape and the preset standard deviation, where n is a non-zero natural number; the second preset key point basic coordinate vector is a vector of the key point coordinates of the n predetermined basic mouth shapes of the preset reference character based on the preset reference mouth shape and the preset standard deviation; the preset weight vector is a vector of the weights of the basic mouth shapes of the two-dimensional character.

[0155] The second processing module 60222 is used to determine that the lip shape indicated by the actual coordinate vector of the key point is the actual lip shape.

[0156] In one example, the device 60 further includes a model training unit 604 , and the model training unit 604 includes an acquisition module 6041 and a training module 6042 .

[0157] Acquisition module 6041 is used to acquire a three-dimensional audio-visual dataset and pre-process the three-dimensional audio-visual dataset to obtain training data for a preset two-dimensional lip prediction model; wherein the three-dimensional audio-visual dataset includes three-dimensional video data and second voice data, and the second voice data is the voice data of a three-dimensional character in the three-dimensional video data; the training data includes input data and calibration data.

[0158] The training module 6042 is used to train the initial model of the preset two-dimensional lip shape prediction model according to the training data until the preset conditions are met, and then output the preset two-dimensional lip shape prediction model.

[0159] In one example, the acquisition module 6041 includes an extraction module 60411 , an alignment module 60412 , and a determination module 60413 .

[0160] The extraction module 60411 is used to extract the character image of the three-dimensional character corresponding to the second voice data in the three-dimensional video data, and to capture a partial character image in the character image; wherein the partial character image includes the lower half of the face below the nose of the three-dimensional character.

[0161] The alignment module 60412 is used to obtain the coordinates of the initial key points of the lower half of the face in the local character image, align the lower half of the face in the local character image, obtain the aligned local character image, and determine the real key point coordinate vector of the aligned local character image.

[0162] The determination module 60413 is used to determine the second speech data as input data and determine the real key point coordinate vector as calibration data.

[0163] In one example, the alignment module 60412 is specifically used to determine the center point and center of mass of the lower half of the face based on the coordinates of the initial key points; and repeatedly adjust the coordinates of the initial key points based on the center point and center of mass of the lower half of the face and the center point and center of mass of the preset average face until the lower half of the face is aligned with the preset average face.

[0164] In one example, the training module 6042 includes an input module 60421 , a calculation module 60422 , and an output module 60423 .

[0165] Input module 60421 is used to input the second voice data into the initial model of the preset two-dimensional lip shape prediction model and output a predicted key point coordinate vector; wherein the predicted key point coordinate vector is the key point coordinate vector of the lip shape of the preset reference character when interpreting the second voice data.

[0166] The calculation module 60422 is used to bring the predicted key point coordinate vector and the real key point coordinate vector into the preset loss function to determine the loss value.

[0167] The output module 60423 is used to adjust the model parameters of the initial model according to the loss value, and determine that the model when the loss value is the target value is the preset two-dimensional lip shape prediction model.

[0168] The device provided in this embodiment can be used to execute the method of the above embodiment. Its implementation principle and technical effects are similar and will not be described in detail here.

[0169] It should be noted that it should be understood that the division of the various modules of the above device is merely a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. Moreover, these modules can all be implemented in the form of software called by processing elements; they can also all be implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. In addition, they can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The implementation of other modules is similar. In addition, these modules can all or partly be integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed by the hardware integrated logic circuit in the processor element or by instructions in the form of software.

[0170] Figure 7 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. As shown in Figure 7, the electronic device 70 includes: a processor 701 and a memory 702 communicatively connected to the processor.

[0171] The memory 702 stores computer-executable instructions; the processor 701 executes the computer-executable instructions stored in the memory 702 to implement any of the aforementioned methods.

[0172] In the specific implementation of the above-mentioned electronic device, it should be understood that the processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or any conventional processor. The method disclosed in conjunction with the embodiments of the present application can be directly implemented as a hardware processor, or can be implemented by a combination of hardware and software modules in the processor.

[0173] An embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement any of the aforementioned methods.

[0174] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented by hardware associated with computer instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0175] An embodiment of the present application further provides a computer program product, comprising a computer program, which is used to implement any of the aforementioned methods when executed by a processor.

[0176] An embodiment of the present application also provides a computer program, which, when executed by a processor, is used to implement any of the aforementioned methods.

[0177] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, is used to implement any of the above-mentioned methods for controlling active heating of electric drives.

[0178] An embodiment of the present application also provides a computer program, which implements any of the aforementioned methods when executed by a processor.

[0179] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0180] It should be understood that the present application is not limited to the exact structure described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

[0181] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. An animation display method based on voice-driven two-dimensional lip animation, characterized in that: The method comprises: Inputting first voice data into a preset two-dimensional lip shape prediction model and outputting predicted lip shape; wherein the first voice data is voice data of a two-dimensional character in a two-dimensional lip shape animation, and the predicted lip shape is the lip shape of a preset reference character in the preset two-dimensional lip shape prediction model when performing the first voice data; Redirecting the predicted lip shape to determine the actual lip shape of the two-dimensional character when performing the first voice data; When playing the first voice data, the lip shape of the two-dimensional character in the two-dimensional lip animation is displayed as the actual lip shape.

2. The method according to claim 1, characterized in that Redirecting the predicted lip shape to determine the actual lip shape of the two-dimensional character when performing the first voice data includes: Determine a key point coordinate displacement vector of the predicted lip shape relative to the preset reference lip shape according to the key point coordinate vector of the predicted lip shape and the key point coordinate vector of the preset reference lip shape of the preset reference character; Determine the actual coordinate vector of the key point of the lip shape of the two-dimensional character when performing the first voice data according to the key point coordinate displacement vector and the key point coordinate vector of the preset reference lip shape of the two-dimensional character; The lip shape indicated by the actual coordinate vector of the key point is determined as the actual lip shape.

3. The method according to claim 1, characterized in that Redirecting the predicted lip shape to determine the actual lip shape of the two-dimensional character when performing the first voice data includes: Determine the actual coordinate vector of the key points of the mouth shape of the two-dimensional character when interpreting the first voice data according to the first preset key point basic coordinate vector, the second preset key point basic coordinate vector, the key point coordinate vector of the predicted mouth shape, and the preset weight vector; wherein the first preset key point basic coordinate vector is a vector of the key point coordinates of n predetermined two-dimensional character basic mouth shapes based on the preset reference mouth shape and the preset standard deviation of the two-dimensional character, and n is a non-zero natural number; the second preset key point basic coordinate vector is a vector of the key point coordinates of n predetermined reference character basic mouth shapes based on the preset reference mouth shape and the preset standard deviation of the preset reference character; the preset weight vector is a vector of the weight of the basic mouth shape of the two-dimensional character; The lip shape indicated by the actual coordinate vector of the key point is determined as the actual lip shape.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: Acquire a three-dimensional audio-visual data set, and pre-process the three-dimensional audio-visual data set to obtain training data for the preset two-dimensional lip prediction model; wherein the three-dimensional audio-visual data set includes three-dimensional video data and second voice data, and the second voice data is the voice data of the three-dimensional character in the three-dimensional video data; the training data includes input data and calibration data; The initial model of the preset two-dimensional lip shape prediction model is trained according to the training data until a preset condition is met, and the preset two-dimensional lip shape prediction model is output.

5. The method according to claim 4, characterized in that Preprocessing the three-dimensional audio-visual data set to obtain training data for the preset two-dimensional lip shape prediction model includes: Extracting a character image of a three-dimensional character corresponding to the second voice data in the three-dimensional video data, and intercepting a partial character image in the character image; wherein the partial character image includes the lower half of the face below the nose of the three-dimensional character; Obtaining coordinates of initial key points of the lower half of the face in the partial character image, performing alignment processing on the lower half of the face in the partial character image, obtaining an aligned partial character image, and determining a real key point coordinate vector of the aligned partial character image; The second speech data is determined as the input data, and the real key point coordinate vector is determined as the calibration data.

6. The method according to claim 5, characterized in that Performing alignment processing on the lower half of the face in the partial character image, including: Determine the center point and the centroid of the lower half of the face according to the coordinates of the initial key points; According to the center point and the centroid of the lower half of the face and the center point and the centroid of the preset average human face, the coordinates of the initial key points are repeatedly adjusted until the lower half of the face is aligned with the preset average human face.

7. The method according to claim 5, characterized in that Training an initial model of the preset two-dimensional lip shape prediction model according to the training data until a preset condition is met, and outputting the preset two-dimensional lip shape prediction model, comprising: Input the second voice data into the initial model of the preset two-dimensional lip shape prediction model, and output a predicted key point coordinate vector; wherein the predicted key point coordinate vector is a key point coordinate vector of the lip shape of the preset reference character when performing the second voice data; Substitute the predicted key point coordinate vector and the real key point coordinate vector into a preset loss function to determine a loss value; According to the loss value, the model parameters of the initial model are adjusted, and the model when the loss value is determined to be the preset two-dimensional lip shape prediction model.

8. An animation display device based on voice-driven two-dimensional lip animation, characterized in that: The device comprises: A prediction unit, configured to input first voice data into a preset two-dimensional lip shape prediction model and output a predicted lip shape; wherein the first voice data is voice data of a two-dimensional character in a two-dimensional lip shape animation, and the predicted lip shape is the lip shape of a preset reference character in the preset two-dimensional lip shape prediction model when performing the first voice data; A redirection unit, configured to redirect the predicted lip shape to determine the actual lip shape of the two-dimensional character when performing the first voice data; The display unit is used to display the lip shape of the two-dimensional character in the two-dimensional lip animation as the actual lip shape when playing the first voice data.

9. An electronic device, characterized in that: The electronic device comprises: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

11. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 7 when being executed by a processor.

12. A computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Voice-based mouth shape animation synthesis device and method and readable storage medium

    CN108763190A

  • Prediction model training method and device, mouth shape animation generation method and device, equipment and medium

    CN115602157A

  • Animation display method and device based on voice-driven two-dimensional mouth shape animation

    CN116543067A

  • Speech and text driven HMM-based body animation synthesis

    US20100082345A1

Cited By

  • Portrait action driving method and device, storage medium and computer equipment

    CN119540416A