A facial animation generation method, system and storage medium

Abstract

The present invention discloses a method, system and storage medium for generating facial animation, and the method comprises the following steps: receiving audio information and text information; generating a plurality of lip shapes according to the audio information; calculating the lip shape change rate of the audio information; generating lip shape animation according to the lip shape change rate and a plurality of the lip shapes; obtaining the global expression of the text information and the weights of a plurality of preset expressions in each part of the text information; generating expression animation according to the weights of the global expression and a plurality of preset expressions in each part of the text information; generating facial animation according to the lip shape animation and the expression animation. The present invention introduces the lip shape change rate as a reference quantity to generate different lip shape animations, generates facial animations by combining different lip shape animations with different expressions, and refers to the influence of each line of lyrics in a song on emotional expression, so that the facial animation of a virtual person when singing is richer and more natural.

Description

Technical Field

[0001] The invention patent relates to the field of animation technology, and in particular to a method, system and storage medium for generating facial animation. Background Art

[0002] With the popularization of virtual human (digital human) technology, virtual humans are widely used in various fields, mainly divided into two types: virtual anchors who mainly speak and virtual singers who mainly sing and dance. In the field of virtual human research, facial animation synthesis has always been a research focus. There are currently two main methods: traditional face capture methods and methods that use machine learning or deep learning algorithms for synthesis.

[0003] The traditional facial capture method is to capture the facial movements of actors through facial capture software, and then the facial movements are refined by animators in the later stage. This method has good results, but it has two shortcomings: first, it is inefficient and requires a large number of animators to make manual adjustments, which is time-consuming and expensive to produce; second, it is heavily dependent on the performance of the motion capture actors. Each song must be recorded by a motion capture actor separately, and the quality is difficult to guarantee due to the performance level of the facial capture actors.

[0004] A synthesis method using machine learning or deep learning algorithms. Lip shape generation obtains the time information of phonemes through lip shape alignment, and then automatically generates facial animation using lip shape fusion. Expression generation mainly controls the changes of eyes and lips through Blendshape, and fuses the expressions to generate facial expressions. Although this method is efficient, it is difficult to take into account the performance details of lip shape and expression and the influence of multiple factors, and the synthesized animation has a strong mechanical feel.

[0005] At present, the two methods have been applied to the virtual anchor speech scene, but the virtual singing represented by the virtual singer is still mainly based on the traditional face capture method, and there is no algorithm specifically for the automatic synthesis of lip shape and expression when singing. The lip shape synthesis of singing and speaking is very different, and the algorithm of lip shape and expression of speaking cannot be directly transplanted to singing. The main differences include:

[0006] Singing has a wide range of speech speeds. For example, rap songs are much faster than speaking, but ballads are much slower than speaking. For extremely fast and slow speech speeds, algorithms designed for speaking are not suitable.

[0007] The expressions when singing are different from those when speaking. When singing, there are more artistic expressions such as affection and looking into the distance, some of which are very exaggerated, while the expressions when speaking are more daily expressions.

[0008] Therefore, in view of the wide range of singing speeds and the differences in singing expressions from speaking, a facial animation generation method is needed to solve the problem that traditional lip shape and expression algorithms cannot be applied, so as to efficiently synthesize high-quality singing facial animations, thereby meeting the huge demand for virtual human singing and dancing performances in the entertainment and animation markets. Summary of the invention

[0009] The purpose of the present invention is to provide a facial animation generation method, system and storage medium to solve the problem that the existing virtual human facial animation generation algorithm model cannot be directly applied to the field of virtual human singing, thereby effectively generating virtual human facial animation with more realistic and vivid lip shape and expression.

[0010] In order to achieve the above invention objects, the present invention is implemented by the following technical solutions:

[0011] The present invention provides a method for generating facial animation, comprising the steps of:

[0012] Receive audio and text messages;

[0013] generating a plurality of lip shapes according to the audio information;

[0014] Calculating the lip shape change rate of the audio information;

[0015] Generate a lip shape animation according to the lip shape change rate and a plurality of lip shapes;

[0016] Obtaining the global expression of the text information and the weights of several preset expressions of each part of the text information;

[0017] Generate an expression animation according to the weights of the global expression and a plurality of preset expressions in each part of the text information;

[0018] Generate facial animation according to the lip animation and the facial expression animation.

[0019] The present invention introduces the lip shape change rate as a reference quantity to generate different lip shape animations. Different lip shape animations are combined with different expressions to generate facial animations. This can solve the impact of different speaking speeds on facial expressions when a virtual person sings, as well as the impact of each line of lyrics in a reference song on emotional expressions, making the facial animation of the virtual person when singing more rich and natural.

[0020] Furthermore, the present invention also provides a method for generating facial animation, wherein the method of generating a plurality of lip shapes according to the audio information specifically comprises:

[0021] Convert each part of the audio information into a number of phonemes;

[0022] Generate a corresponding first mouth shape according to each of the phonemes;

[0023] Respectively obtain a number of complete pronunciation time periods of each part of the audio information;

[0024] Time-aligning each of the complete pronunciation periods with the corresponding phonemes;

[0025] The first mouth shape corresponding to the phoneme in the middle position of each complete pronunciation period is identified as the key mouth shape.

[0026] The present invention introduces two lip shapes for generating lip shape animations under different lip shape change rates, so that the virtual person can generate more vivid and natural facial expressions when singing songs with different speaking speeds.

[0027] Furthermore, the present invention provides a method for generating facial animation, wherein the method for generating the lip animation according to the lip shape change rate and a plurality of the lip shapes specifically comprises:

[0028] Generate a plurality of lip animations according to the lip shape change rate and a plurality of lip shapes corresponding to each part of the audio information;

[0029] Combining several segments of the sub-lip animations in order to generate the lip animation;

[0030] The step of generating a plurality of segment lip animations according to the lip shape change rate and the lip shapes corresponding to the respective parts in the audio information specifically includes:

[0031] When the mouth shape change rate is less than a preset threshold, generating the sub-mouth shape animation according to a plurality of the first mouth shapes corresponding to each part of the audio information;

[0032] When the lip shape change rate is greater than a preset threshold, the sub-lip shape animation is generated according to a plurality of the key lip shapes corresponding to each part of the audio information.

[0033] Furthermore, the present invention provides a method for generating facial animation, wherein the step of obtaining the global expression of the text information specifically comprises:

[0034] Calculating the weights of several preset emotions in the text information according to a preset emotion classification model, and respectively calculating the weights of several preset emotions in each part of the text information;

[0035] Converting a plurality of the preset emotions into corresponding preset expressions according to a preset expression library;

[0036] The preset expression with the highest weight in the text information is identified as the global expression.

[0037] Furthermore, the present invention provides a method for generating facial animation, wherein the method generates facial animation according to the weights of the global expression and a plurality of preset expressions in each part of the text information, specifically comprising:

[0038] Combining the global expression and a plurality of the preset expressions to generate a plurality of local expressions by using a weighted average method;

[0039] A plurality of the local expressions are combined in sequence to generate the expression animation.

[0040] The present invention combines multiple expressions by weighted average processing to generate expression animation, so that when the virtual person sings and the song has emotional changes, the facial expressions of the virtual person are more varied and natural, which is in line with the expression of the song's emotions.

[0041] Furthermore, the present invention provides a method for generating facial animation, characterized in that after generating the lip animation according to the lip shape change rate and the lip shape, and before obtaining the global expression of the text information and the weights of several preset expressions of each part of the text information, the method further comprises:

[0042] Using a linear interpolation method to linearly smooth the lip animation;

[0043] Define the Gaussian smoothing window, the formula is as follows:

[0044]

[0045] in k is the lip shape change rate, i is the relative time scale of the audio information, and e is a constant;

[0046] The lip animation is subjected to nonlinear smoothing processing, and the formula is as follows:

[0047]

[0048] in is the mouth shape at the current moment after the nonlinear smoothing process, N is the smoothing window width, N=3f s / k,f s is the audio sampling frequency.

[0049] The present invention adopts nonlinear smoothing processing in the process of mouth shape change, so as to avoid mechanical changes of the mouth shape of the virtual person when there is dragging sound during singing, and make the facial expression of the virtual person more natural and smooth during singing.

[0050] Furthermore, the present invention provides a method for generating facial animation, after generating a plurality of lip shapes according to the audio information and before generating lip shape animation according to the lip shape change rate and the plurality of lip shapes, the method further comprises:

[0051] Calculate the first average energy of the audio information within M seconds, the formula is as follows:

[0052]

[0053] where e t is the first average energy, t is the current time of the audio information, and the unit of t is minutes;

[0054] The modulation ratio is calculated as the second average energy after regularization, and the formula is as follows:

[0055]

[0056] Where r t The second average energy, is the average energy vector of the audio information within 1 to T seconds;

[0057] The lip shape is adjusted according to the second average energy.

[0058] The present invention can improve the matching degree between the lip shape and the song by modulating the amplitude of the lip shape of the virtual person during singing according to the volume, so that the facial expression of the virtual person during singing is closer to the song.

[0059] In addition, the present invention also provides a facial animation generation system, comprising:

[0060] A receiving module, used for receiving audio information and text information, wherein the text information includes a plurality of sub-text information;

[0061] A lip shape generation module, connected to the receiving module, for generating a plurality of lip shapes according to the audio information;

[0062] A rate calculation module, connected to the receiving module, for calculating the rate of change of the outlet pattern according to the audio information;

[0063] A lip shape animation generation module, connected to the lip shape generation module and the rate calculation module, for generating a lip shape animation according to the lip shape change rate and a plurality of lip shapes;

[0064] An acquisition module, connected to the receiving module, for acquiring the global expression of the text information and the weights of several preset expressions of various parts in the text information;

[0065] An expression animation generation module generates an expression animation according to the weights of the global expression and a plurality of preset expressions in each part of the text information;

[0066] A facial animation generation module is used to generate facial animation according to the lip animation and the expression animation.

[0067] The present invention introduces the lip shape change rate as a reference quantity to generate different lip shape animations. Different lip shape animations are combined with different expressions to generate facial animations. This can solve the impact of different speaking speeds on facial expressions when a virtual person sings, as well as the impact of each line of lyrics in a reference song on emotional expressions, making the facial animation of the virtual person when singing more rich and natural.

[0068] Furthermore, the present invention provides a facial animation generation system, further comprising:

[0069] A smoothing module, connected to the first animation generation module, for performing nonlinear smoothing processing on the lip animation;

[0070] an amplitude adjustment module, connected to the lip shape generation module, for adjusting the amplitudes of a plurality of the lip shapes according to the energy of the audio information;

[0071] The smoothing module includes a linear smoothing unit, a smoothing window definition unit and a nonlinear smoothing unit.

[0072] A linear smoothing unit, used for linearly smoothing the lip animation by using a linear interpolation method;

[0073] The smoothing window definition unit is used to define a Gaussian smoothing window. The formula is as follows:

[0074]

[0075] in k is the lip shape change rate, i is the relative time scale of the audio information, and e is a constant;

[0076] The nonlinear smoothing unit is used to perform nonlinear smoothing processing on the lip animation. The formula is as follows:

[0077]

[0078] in is the mouth shape at the current moment after nonlinear smoothing, N is the smoothing window width, N = 3f s / k,f s is the audio sampling frequency.

[0079] The amplitude adjustment module includes a first average energy calculation unit, a second average energy calculation unit and a lip adjustment unit.

[0080] The first average energy calculation unit is used to calculate the first average energy of the audio information from time t to time t1, and the formula is as follows:

[0081]

[0082] where e t is the first average energy, t is the current time of the audio information, and the unit of t is minutes;

[0083] The second average energy calculation unit is used to calculate the second average energy after the modulation ratio is regularized, and the formula is as follows:

[0084]

[0085] where r t is the second average energy, is the average energy vector of the audio information within 1 to T seconds;

[0086] The lip shape adjustment unit is used to adjust the lip shape according to the second average energy.

[0087] In addition, the present invention also provides a storage medium, in which at least one instruction is stored, and the instruction is used to implement the operations performed by the facial animation generation method as described above.

[0088] The present invention provides a method, system and storage medium for generating facial animation, which have at least the following gain effects:

[0089] 1) The present invention introduces the mouth shape change rate as a reference quantity to generate different mouth shape animations. Different mouth shape animations are combined with different expressions to generate facial animations, which can solve the influence of different speech speeds on facial expressions when the virtual person sings, and the influence of each line of lyrics in the reference song on emotional expressions, so that the facial animation of the virtual person when singing is richer and more natural;

[0090] 2) The present invention introduces two lip shapes for generating lip shape animations at different lip shape change rates, so that the virtual person can generate more vivid and natural facial expressions when singing songs at different speech speeds;

[0091] 3) The present invention combines multiple expressions by weighted average processing to generate expression animation, so that when the virtual person sings and the song has emotional changes, the facial expressions of the virtual person are more varied and natural, which is consistent with the expression of the song's emotions;

[0092] 4) The present invention adopts nonlinear smoothing processing in the process of mouth shape change to avoid mechanical changes in the mouth shape of the virtual person when there is a dragging phenomenon during singing, so that the facial expression of the virtual person during singing is more natural and smooth;

[0093] 5) The present invention modulates the amplitude of the lip shape of the virtual person during singing according to the volume, which can improve the matching degree between the lip shape and the song, so that the facial expression of the virtual person during singing is closer to the song;

[0094] 6) The present invention can automatically generate facial animations including lip shapes and expressions based only on the acquired target song and its lyrics, without human intervention, and can efficiently produce a large number of virtual human singing animations. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] The following will explain the preferred implementation mode in a clear and understandable manner with reference to the accompanying drawings, and further illustrate the above-mentioned characteristics, technical features, advantages and implementation methods of a facial animation generation method, system and storage medium.

[0096] Figure 1 is a flow chart of a method for generating facial animation of the present invention;

[0097] Figure 2 It is a flow chart of a method for generating lip shapes in a method for generating facial animation of the present invention;

[0098] Figure 3 It is a flow chart of a method for generating lip animation in a method for generating facial animation of the present invention;

[0099] Figure 4 It is a flow chart of a method for obtaining global expression in a method for generating facial animation of the present invention;

[0100] Figure 5 It is a flow chart of a facial expression animation generation method in a facial animation generation method of the present invention;

[0101] Figure 6 is another flow chart of a method for generating facial animation of the present invention;

[0102] Figure 7 is another flow chart of a method for generating facial animation of the present invention;

[0103] Figure 8 is a schematic diagram of a facial animation generation system of the present invention;

[0104] Fig. 9 It is a schematic diagram of a lip shape generation module in a facial animation generation system of the present invention;

[0105] Fig.10 It is a schematic diagram of a lip animation generation module in a facial animation generation system of the present invention;

[0106] Fig.11 is a schematic diagram of an acquisition module in a facial animation generation system of the present invention;

[0107] Fig.12 It is a schematic diagram of an expression animation generation module in a facial animation generation system of the present invention;

[0108] Fig.13 is a schematic diagram of a smoothing module in a facial animation generation system of the present invention;

[0109] Fig.14 It is a schematic diagram of an amplitude adjustment module in a facial animation generation system of the present invention;

[0110] Numbers in the figure: 10-receiving module, 20-lip shape generation module, 30-rate calculation module, 40-lip shape animation generation module, 50-acquisition module, 60-expression animation generation module, 70-facial animation generation module, 80-smoothing module, 90-amplitude adjustment module, 21-phoneme conversion unit, 22-first lip shape generation unit, 23-complete pronunciation period acquisition unit, 24-time alignment unit, 25-key lip shape recognition unit 25, 41-judgment unit, 42-sub-lip shape animation generation unit, 43-lip shape animation combination unit, 51-emotion classification model establishment unit, 52-expression library establishment unit, 53-weight calculation unit, 54-expression conversion unit, 55-global expression recognition unit, 61-local expression generation unit, 62-expression animation composition unit, 81-linear smoothing unit, 82-smoothing window definition unit, 83-nonlinear smoothing unit, 91-first average energy calculation unit, 92-second average energy calculation unit, 93-lip shape debugging unit. DETAILED DESCRIPTION

[0111] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0112] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections.

[0113] In order to simplify the drawings, only the parts related to the present invention are schematically shown in each figure, and they do not represent the actual structure of the product. In addition, in order to simplify the drawings and facilitate understanding, in some figures, only one of the parts with the same structure or function is schematically drawn or marked. In this article, "one" not only means "only one", but also means "more than one".

[0114] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0115] In addition, in the description of the present application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0116] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the specific implementation methods of the present invention will be described below with reference to the accompanying drawings. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings and other implementation methods can be obtained based on these drawings without creative work.

[0117] Example 1

[0118] One embodiment of the present invention, as Figure 1 As shown, the present invention provides a method for generating facial animation, comprising the steps of:

[0119] S100 receives audio information and text information.

[0120] Specifically, audio information includes singing audio, reciting audio, etc., mainly referring to the singing audio that needs to generate facial expressions during the virtual human singing process; text information includes lyrics, reciting manuscripts, etc., mainly referring to the lyrics that need to generate facial expressions during the virtual human singing process.

[0121] S200 generates a plurality of lip shapes according to the audio information.

[0122] Specifically, methods for generating lip shapes include traditional facial capture methods, methods using deep learning algorithms such as machine learning, and methods for converting audio into phonemes and generating lip shapes, etc.

[0123] S300 calculates the lip shape change rate of the audio information.

[0124] Specifically, the lip shape change rate refers to how fast the lip shape changes over time. It is estimated by referring to the speaking speed and is defined as the average number of lyrics per second. Usually, fast songs change lip shape quickly, while slow songs change slowly. This rate helps to globally set the speed of lip shape changes and control the degree of lip shape.

[0125] S400 generates lip animation according to the lip shape change rate and a plurality of lip shapes.

[0126] Specifically, the mouth shape change speed is set globally and the degree of mouth shape is controlled according to the mouth shape change rate. When the mouth shape change rate is large, the switching speed between each mouth shape is fast, and there is a certain degree of jitter in the switching of each mouth shape; when the mouth shape rate is small, the switching speed is slow, the transition is smooth, and the jitter is less.

[0127] S500 obtains the global expression of the text information and the weights of several preset expressions of various parts in the text information.

[0128] Specifically, the expression generation in the present invention mainly includes two parts, global expression and local expression. The average emotion type of the whole song determines the global expression of the whole song, and at the same time, the emotion of each lyric will affect the change of local expression.

[0129] S600 generates expression animation according to the weights of the global expression and several preset expressions in each part of the text information.

[0130] Specifically, the present invention adopts a combination of global basic expressions and local variable expressions. The global expression adopts a winner-takes-all strategy, and the local expression adopts a weighted average strategy. The combination of the two produces an overall facial expression that has a clear emotional style and rich in changeable emotional expressions.

[0131] S700 generates facial animation based on lip animation and expression animation.

[0132] Specifically, the final facial animation is achieved by fusing lip animation and expression animation, and the expression animation is converted from the resolution of the lyrics to the temporal resolution consistent with the lip animation. t , the fused animation is:

[0133] f t = l t +q t ,

[0134] where f t represents the model parameters of the generated facial animation, l t Represents the model parameters of the generated lip animation, q t Represents the model parameters of the generated expression animation.

[0135] Specifically, in this embodiment, the order of generating lip animation and generating facial expression animation is not completely limited. In the actual facial expression generation process, the lip animation can be generated first and then the facial expression animation, or the facial expression animation can be generated first and then the lip animation. The generation order of the lip animation and the facial expression animation has no effect on the specific generation steps.

[0136] Specifically, in this embodiment, the order of generating a number of lip shapes according to the audio information and calculating the rate of change of the lip shapes of the audio information is not limited. In the actual facial expression generation process, you can first generate a number of lip shapes according to the audio information and then calculate the rate of change of the lip shapes of the audio information, or you can first calculate the rate of change of the lip shapes of the audio information and then generate a number of lip shapes according to the audio information. The order of calculating the rate of change of the lip shapes and generating the lip shapes according to the audio information has no effect on the specific generation steps.

[0137] This embodiment proposes a method for synthesizing facial animation of a virtual human singing, filling the gap of a complete algorithm for facial synthesis of virtual human singing. The method proposed in this embodiment synthesizes and fuses the two most important elements of the face, namely the lip shape and expression, respectively, to generate rich, realistic and varied facial animations. This embodiment focuses on the application scenario of virtual human singing and proposes a complete solution. In this embodiment, the influence of different speech rates on facial expression animation is considered when the virtual human has a wide range of speaking rates when singing, and the facial expression changes are generated according to the different speaking rates during the singing process, as well as the influence of each line of lyrics on the facial expression. The facial animation of the virtual human during the singing process is comprehensively generated, making the facial expression of the virtual human richer and more natural when singing.

[0138] Example 2

[0139] Based on the facial animation generation method in Example 1, Figure 2 As shown, several lip shapes are generated according to the text information, including:

[0140] S210 converts each part of the audio information into a plurality of phonemes.

[0141] Specifically, in this embodiment, pinyin is divided into the smallest unit and a phoneme library is constructed.

[0142] As shown in Table 1:

[0143] Initials b, p, m, f, d, t, n, l, g, k, h, j, q, x, zh, ch, sh, r, z, c, s, y, w Single vowel a, o, e, i, u, ü, er Final rhyme nn,ng

[0144] Table 1

[0145] The phoneme library contains all initial consonants, single vowels and final consonants, where n and nn represent the pinyin n of the initial consonant and final consonant respectively, and each phoneme corresponds to a mouth shape.

[0146] S220 generates a corresponding first mouth shape according to each phoneme.

[0147] Specifically, the first lip shape refers to identifying the lip shape corresponding to the phoneme information, breaking down each song into parts divided by each line of lyrics, each line of lyrics is divided into multiple phonemes, and each phoneme corresponds to a lip shape.

[0148] S230 respectively obtains a number of complete pronunciation time periods of each part of the audio information.

[0149] Specifically, each completely pronounced character or word in each line of lyrics is obtained as a complete pronunciation period.

[0150] S240 performs time alignment on each complete pronunciation period and the corresponding phoneme.

[0151] Specifically, each fully pronounced character or word is matched with multiple phonemes corresponding to the character or word in time. That is, during the singing process of the song, the time required for singing a single character is obtained, and the multiple phonemes contained in the character are matched to express the time required for the mouth shapes corresponding to these phonemes. Alignment methods include traditional alignment based on Viterbi decoding in HMM and alignment based on TDNN neural network.

[0152] S250 identifies the first mouth shape corresponding to the phoneme in the middle of each complete pronunciation period as the key mouth shape.

[0153] Specifically, the second mouth shape refers to the key mouth shape in the pronunciation process, that is, the most prominent mouth shape during pronunciation. It is usually in the middle position of the complete pronunciation duration of a phoneme and is relatively less affected by the context phonemes.

[0154] Example 3

[0155] Based on the facial animation generation method in Example 2, Figure 3 As shown, the lip animation is generated according to the lip shape change rate and a number of lip shapes, specifically including:

[0156] S410 generates a plurality of lip animations according to the lip shape change rate and a plurality of lip shapes corresponding to each part of the audio information.

[0157] S420 combines several segments of lip-sync animations in order to generate a lip-sync animation.

[0158] Wherein, S410 generates a sub-lip animation according to the lip shape change rate and the corresponding lip shapes, specifically including:

[0159] Determine the relationship between the mouth shape change rate and the preset threshold.

[0160] Specifically, the mouth shape change speed is set globally and the degree of mouth shape is controlled according to the mouth shape change rate. When the mouth shape rate is greater than the preset threshold, and the switching speed between each mouth shape is fast, a certain degree of jitter is allowed; when the mouth shape rate is less than the preset threshold, the switching speed is slow, the transition is smooth, and the jitter is less.

[0161] The switching speed refers to the number of expressions presented per second. The threshold and switching speed are set manually. For example, when the preset threshold is 2 and the switching speed is 2, if the mouth shape change rate is greater than 2, the expression is switched at a speed of two expressions per second.

[0162] When the lip shape change rate is less than a preset threshold, the sub-lip shape animation is generated according to a plurality of first lip shapes corresponding to each part of the audio information.

[0163] Specifically, when the mouth shape changes at a low rate, the mouth shapes corresponding to all phonemes will be displayed to a certain extent, and the switching is slow, the transition is smooth, and the jitter is less. That is to say, when singing slow songs, the switching rate between mouth shapes is slow, and each mouth shape will be fully expressed.

[0164] When the lip shape change rate is greater than a preset threshold, a sub-lip shape animation is generated according to a number of key lip shapes corresponding to each part of the audio information.

[0165] Specifically, when the lip shape changes at a high rate, only the key lip shape is made during the lip shape switching process, and the switching between key lip shapes is very fast, allowing a certain degree of jitter. That is to say, when singing fast songs, the switching rate between lip shapes is fast, only the key lip shape is shown during the lip shape change process, and the fullness of the lip shape will be reduced accordingly.

[0166] This embodiment introduces two lip shapes for generating lip shape animations at different lip shape change rates, so that the virtual person can generate more vivid and natural facial expressions when singing songs at different speaking speeds.

[0167] Example 4

[0168] A method for generating facial animation based on any one of Embodiments 1 to 3, such as Figure 4 As shown, the global expressions of the text information are obtained, including:

[0169] S510 pre-establishes an emotion classification model and an expression library of expressions corresponding to each emotion.

[0170] Specifically, the sentiment classification model uses Text CNN, which performs well in short text semantic classification. The sentiment data required for training includes 73 Chinese songs, and the overall sentiment of each song and the sentiment of each lyric are manually annotated. The Text CNN training is verified using the ten-fold crossover method, and an 88.32% single sentence recognition accuracy rate is obtained.

[0171] In this embodiment, four emotions of the song are defined as joyful, depressed, critical and affectionate. The animator creates a static expression for each emotion, which is recorded as q j ,q d ,q c ,q a , joyful, depressed, critical, and affectionate j ,q d ,q c ,q a Create an expression library.

[0172] S520 calculates the weights of several preset emotions in the text information according to the preset emotion classification model, and respectively calculates the weights of several preset emotions of various parts in the text information.

[0173] Specifically, the weights of the four preset emotions of joy, sadness, criticism, and affection expressed by the lyrics of the entire song are calculated according to the emotion classification model, and the weights of the four preset emotions of joy, sadness, criticism, and affection in each sentence of the lyrics are also calculated.

[0174] S530 converts a plurality of preset emotions into corresponding preset expressions according to a preset expression library.

[0175] S540 identifies the preset expression with the highest weight in the text information as a global expression.

[0176] Specifically, each emotion of the whole song is scored, and the expression corresponding to the emotion with the highest score in the emotion classification of the whole song is recorded as q g .

[0177] Example 5

[0178] Based on the facial animation generation method in Example 4, Figure 5 As shown, the steps of generating expression animation according to the weights of the global expression and several preset expressions in each part of the text information specifically include:

[0179] S610 combines the global expression and several preset expressions to generate several local expressions by using a weighted average method.

[0180] Specifically, the love song and expression weight analysis is performed on each line of the entire song, and the scores of the four emotions of the i-th line of lyrics are The fusion expression of this line of lyrics is:

[0181]

[0182] Among them, w is the proportion of global expression. The higher the proportion, the less local changes; the lower the proportion, the more local emotional fluctuations. Due to the use of expression weighting, a variety of composite expressions and micro-expressions can be combined, such as an expression that is mainly affectionate and slightly sad, thereby increasing the diversity of expressions.

[0183] S620 combines a plurality of partial expressions in sequence to generate an expression animation.

[0184] Specifically, the global expression generated based on the general meaning of the lyrics of the entire song is weighted with the local expressions in each sentence of the lyrics, and then the generated local expressions are animated and synthesized according to the order in which the song is sung.

[0185] This embodiment combines multiple expressions by weighted average processing to generate expression animation, so that when the virtual person's facial expressions change emotionally during the singing of the song, the virtual person's facial expressions are more varied and natural, which is consistent with the expression of the song's emotions.

[0186] Example 6

[0187] Based on the facial animation generation method in any one of the above-mentioned embodiments 1 to 5, as Figure 6 As shown, after generating the lip animation according to the lip shape change rate and the lip shape, before obtaining the global expression of the text information and the weights of several preset expressions of each part of the text information, it also includes:

[0188] S810 uses linear interpolation to perform linear smoothing on lip animation.

[0189] S820 defines a Gaussian smoothing window, the formula is as follows:

[0190]

[0191] in k is the mouth shape change rate, is the parameter of the smoothing window, i is the relative time scale of the audio information, and e is a constant.

[0192] Specifically, k is the mouth shape change rate, which refers to the speed of mouth shape change over time, and is defined as the average number of lyrics per second based on the estimated speaking speed. i is the relative time scale of the audio information, which is a relative time scale, which is added or subtracted from the song time scale, with seconds as the unit of measurement, for example i=100 refers to the parameters of the Gaussian smoothing window of the lip animation corresponding to the song 100 seconds after the current moment.

[0193] S830 performs nonlinear smoothing on lip animation, the formula is as follows:

[0194]

[0195] in is the mouth shape at the current moment after nonlinear smoothing, N is the smoothing window width, that is, the sum of the time of the complete pronunciation period of a single word at time t of the audio information and the time of the two complete pronunciation periods of the single word before and after, N = 3f s / k,f s is the audio sampling frequency.

[0196] Specifically, N is the time required to pronounce the word in the lyrics corresponding to time t in the song, and the sum of the time required to pronounce the two previous and next words.

[0197] This embodiment solves the problem that traditional speech lip shape synthesis usually adopts linear transition, that is, the transition between two pinyin lip shapes is filled by linear interpolation. Since the time interval between two pinyins during speech is usually tens of milliseconds, the stiffness of the lip shape cannot be seen visually. However, there is a frequent phenomenon of dragging when singing, and the interval between two pinyins may last for several seconds. If only linear transition is used, the mechanical change of the lip shape will be clearly observed visually. The present invention adopts nonlinear smoothing processing in the process of lip shape change. Make the facial expression of the virtual person more natural and smooth during singing.

[0198] Example 7

[0199] Based on the facial animation generation method in any one of the above-mentioned embodiments 1 to 6, as Figure 7 As shown, after generating a plurality of lip shapes according to the audio information, and before generating lip shape animation according to the lip shape change rate and the plurality of lip shapes, the method further includes:

[0200] S910 calculates the first average energy of the audio information within M seconds, using the following formula:

[0201]

[0202] where e t is the first average energy, t is the current time of the audio information, and the unit of t is seconds;

[0203] For example, j = 1, 2, 3 ... T, a song is usually 4 to 5 minutes long, and a verse or chorus is about 1 minute long, so a 1-minute window is selected to calculate the average energy, that is, M is equal to 60, and the average energy at time t is:

[0204]

[0205] S920 calculates the modulation ratio as the regularized second average energy, and the formula is as follows:

[0206]

[0207] where r t is the second average energy, is the first average energy vector within 1 to T seconds.

[0208] S930 adjusts the lip shape according to the second average energy.

[0209] Specifically, the formula for adjusting the lip shape is:

[0210]

[0211] in is the modulated lip shape parameter, l t It is the lip shape parameter before debugging.

[0212] This embodiment proposes a solution to the problem that the mouth shape will open or close accordingly during singing. In the verse part of a song, that is, from the beginning to the climax, the singer's singing volume is usually lower than the climax part of the chorus. Correspondingly, the opening degree of the mouth shape in the verse part is also lower than that in the chorus part. The present invention modulates the amplitude of the virtual person's mouth shape during singing according to the volume, which can improve the matching degree of the mouth shape and the song, so that the facial expression of the virtual person during singing is closer to the song.

[0213] Example 8

[0214] One embodiment of the present invention, as Figure 8 As shown, the present invention provides a facial animation generation system, comprising:

[0215] The receiving module 10 is used to receive audio information and text information.

[0216] Specifically, audio information includes singing audio, reciting audio, etc., mainly referring to the singing audio that needs to generate facial expressions during the virtual human singing process; text information includes lyrics, reciting manuscripts, etc., mainly referring to the lyrics that need to generate facial expressions during the virtual human singing process.

[0217] The lip shape generation module 20 is connected to the receiving module and is used to generate a plurality of lip shapes according to the audio information.

[0218] Specifically, methods for generating lip shapes include traditional facial capture methods, methods using deep learning algorithms such as machine learning, and methods for converting audio into phonemes and generating lip shapes, etc.

[0219] Among them, Fig. 9 As shown, the lip shape generation module 20 specifically includes:

[0220] The phoneme conversion unit 21 is used to convert each part of the audio information into a plurality of phonemes.

[0221] Specifically, in this embodiment, pinyin is split into the smallest unit and a phoneme library is constructed. As shown in Table 1, the phoneme library includes all initial consonants, single vowels and finals, where n and nn represent the pinyin n of the initial consonant and final respectively, and each phoneme corresponds to a mouth shape.

[0222] The first lip shape generating unit 22 is connected to the phoneme converting unit 21 and is used to generate a corresponding first lip shape according to each phoneme.

[0223] Specifically, the first lip shape refers to identifying the lip shape corresponding to the phoneme information, breaking down each song into parts divided by each line of lyrics, each line of lyrics is divided into multiple phonemes, and each phoneme corresponds to a lip shape.

[0224] The complete pronunciation period acquisition unit 23 is used to respectively acquire a number of complete pronunciation periods of each part in the audio information.

[0225] Specifically, each completely pronounced character or word in each line of lyrics is obtained as a complete pronunciation period.

[0226] The time alignment unit 24 is connected to the phoneme conversion unit and the complete pronunciation period acquisition unit 23, and is used to perform time alignment between each complete pronunciation period and the corresponding phoneme.

[0227] Specifically, each fully pronounced character or word is matched with multiple phonemes corresponding to the character or word in time. That is, during the singing process of the song, the time required for singing a single character is obtained, and the multiple phonemes contained in the character are matched to express the time required for the mouth shapes corresponding to these phonemes. Alignment methods include traditional alignment based on Viterbi decoding in HMM and alignment based on TDNN neural network.

[0228] The key mouth shape recognition unit 25 is connected to the time alignment unit 24 and the first mouth shape generation unit 22, and is used to recognize that the first mouth shape corresponding to the phoneme in the middle position of each complete pronunciation period is the key mouth shape.

[0229] Specifically, the second mouth shape refers to the key mouth shape in the pronunciation process, that is, the most prominent mouth shape during pronunciation. It is usually in the middle position of the complete pronunciation duration of a phoneme and is relatively less affected by the context phonemes.

[0230] The rate calculation module 30 is connected to the receiving module and is used to calculate the export pattern change rate according to the audio information.

[0231] Specifically, the lip shape change rate refers to how fast the lip shape changes over time. It is estimated by referring to the speaking speed and is defined as the average number of lyrics per second. Usually, fast songs change lip shape quickly, while slow songs change slowly. This rate helps to globally set the speed of lip shape changes and control the degree of lip shape.

[0232] The lip animation generation module 40 is connected to the lip generation module and the rate calculation module, and is used to generate lip animation according to the lip change rate and a plurality of lip shapes.

[0233] Specifically, the speed of the mouth shape change and the degree of the mouth shape are controlled globally according to the mouth shape change rate.

[0234] Among them, Fig.10 As shown, the lip animation generation module 40 specifically includes:

[0235] The judging unit 41 is used to judge the relationship between the mouth shape change rate and a preset threshold.

[0236] Specifically, the mouth shape change speed is set globally and the degree of mouth shape is controlled according to the mouth shape change rate. When the mouth shape rate is greater than the preset threshold, and the switching speed between each mouth shape is fast, a certain degree of jitter is allowed; when the mouth shape rate is less than the preset threshold, the switching speed is slow, the transition is smooth, and the jitter is less.

[0237] The switching speed refers to the number of expressions presented per second. The threshold and switching speed are set manually. For example, when the preset threshold is 2 and the switching speed is 2, if the mouth shape change rate is greater than 2, the expression is switched at a speed of two expressions per second.

[0238] The sub-lip animation generating unit 42 is connected to the judging unit 41, and is used to generate a plurality of sub-lip animations according to a plurality of first lip shapes corresponding to each part of the audio information when the lip shape change rate is less than a preset threshold. When the lip shape change rate is greater than the preset threshold, a plurality of sub-lip animations are generated according to a plurality of key lip shapes corresponding to each part of the audio information.

[0239] The lip-sync animation combining unit 43 is connected to the sub-lip-sync animation generating unit 42, and is used to combine several sub-lip-sync animations in sequence to generate a lip-sync animation.

[0240] The acquisition module 50 is connected to the receiving module and is used to acquire the global expression of the text information and the weights of several preset expressions of various parts in the text information.

[0241] Specifically, expression generation in this embodiment mainly includes two parts, global expression and local expression. The average emotion type of the whole song determines the global expression of the whole song. At the same time, the emotion of each line of lyrics will affect the local expression change. Obtaining the global expression of text information means obtaining the global emotion of the whole song according to the general meaning of the lyrics, and then converting the global emotion into a global expression.

[0242] Among them, Fig.11 As shown, the acquisition module 50 specifically includes:

[0243] The sentiment classification model building unit 51 is used to build a sentiment classification model in advance.

[0244] Specifically, the sentiment classification model uses Text CNN, which performs well in short text semantic classification. The sentiment data required for training includes 73 Chinese songs, and the overall sentiment of each song and the sentiment of each lyric are manually annotated. The Text CNN training is verified using the ten-fold crossover method, and an 88.32% single sentence recognition accuracy rate is obtained.

[0245] The expression library building unit 52 is used to build an expression library according to the expressions corresponding to various emotions.

[0246] Specifically, the four emotions of the song are defined as joyful, depressed, critical, and affectionate. The animator creates a static expression for each emotion, which is denoted by q j ,q d ,q c ,q a , joyful, depressed, critical, and affectionate j ,q d ,q c ,q a Create an expression library.

[0247] The weight calculation unit 53 is connected to the emotion classification model establishment unit 51, and is used to calculate the weights of several preset emotions in the text information according to the preset emotion classification model, and to calculate the weights of several preset emotions in each part of the text information respectively.

[0248] Specifically, the weights of the four preset emotions of joy, sadness, criticism, and affection expressed by the lyrics of the entire song are calculated according to the emotion classification model, and the weights of the four preset emotions of joy, sadness, criticism, and affection in each sentence of the lyrics are also calculated.

[0249] The expression conversion unit 54 is connected to the weight calculation unit 53 and the expression library establishment unit 52, and is used to convert a plurality of preset emotions into corresponding preset expressions according to the preset expression library.

[0250] The global expression recognition unit 55 is connected to the expression conversion unit 54 and is used to recognize the preset expression with the highest weight in the text information as the global expression.

[0251] Specifically, each emotion of the whole song is scored, and the expression corresponding to the emotion with the highest score in the emotion classification of the whole song is recorded as the global expression q g .

[0252] The expression animation generation module 60 is connected to the acquisition module 50 and is used to generate expression animation according to the weights of the global expression and several preset expressions in each part of the text information.

[0253] Specifically, the present invention adopts a combination of global basic expressions and local variable expressions. The global expression adopts a winner-takes-all strategy, and the local expression adopts a weighted average strategy. The combination of the two produces an overall facial expression that has a clear emotional style and rich in changeable emotional expressions.

[0254] Among them, Fig.12 As shown, the facial expression animation generation module 60 specifically includes:

[0255] The local expression generating unit 61 is used to generate a plurality of local expressions by combining the global expression and a plurality of preset expressions using a weighted average method.

[0256] Specifically, the love song and expression weight analysis is performed on each line of the entire song, and the scores of the four emotions of the i-th line of lyrics are The fusion expression of this line of lyrics is:

[0257]

[0258] Among them, w is the proportion of global expression. The higher the proportion, the less local changes; the lower the proportion, the more local emotional fluctuations. Due to the use of expression weighting, a variety of composite expressions and micro-expressions can be combined, such as an expression that is mainly affectionate and slightly sad, thereby increasing the diversity of expressions.

[0259] The expression animation composition unit 62 is connected to the local expression generation unit 61 and is used to combine several local expressions in sequence to generate an expression animation.

[0260] Specifically, the global expression generated based on the general meaning of the lyrics of the entire song is weighted with the local expressions in each sentence of the lyrics, and then the generated local expressions are animated and synthesized according to the order in which the song is sung.

[0261] The facial animation generation module 70 is connected to the expression animation generation module 60 and the lip animation generation module 40, and is used to generate facial animation according to the lip animation and the expression animation.

[0262] Specifically, the final facial animation is achieved by fusing lip animation and expression animation, and the expression animation is converted from the resolution of the lyrics to the temporal resolution consistent with the lip animation. t , the fused animation is:

[0263] f t = l t +q t ,

[0264] where f t represents the model parameters of the generated facial animation, l t Represents the model parameters of the generated lip animation, q t Represents the model parameters of the generated expression animation.

[0265] The smoothing module 80 is connected to the lip animation generation module and is used to perform nonlinear smoothing processing on the lip animation.

[0266] Specifically, Fig.13 As shown, the smoothing module 80 includes:

[0267] The linear smoothing unit 81 is used to perform linear smoothing on the lip animation by using a linear interpolation method.

[0268] The smoothing window definition unit 82 is connected to the linear smoothing unit 81 and is used to define a Gaussian smoothing window. The formula is as follows:

[0269]

[0270] in k is the mouth shape change rate, is the parameter of the smoothing window, i is the relative time scale of the audio information, and e is a constant.

[0271] Specifically, k is the mouth shape change rate, which refers to the speed of mouth shape change over time, and is defined as the average number of lyrics per second based on the estimated speaking speed. i is the relative time scale of the audio information, which is a relative time scale, which is added or subtracted from the song time scale, with seconds as the unit of measurement, for example i=100 refers to the parameters of the Gaussian smoothing window of the lip animation corresponding to the song 100 seconds after the current moment.

[0272] The nonlinear smoothing unit 83 is connected to the smoothing window definition unit 82 and is used to perform nonlinear smoothing processing on the lip animation. The formula is as follows:

[0273]

[0274] in is the mouth shape at the current moment after nonlinear smoothing, N is the smoothing window width, that is, the sum of the time of the complete pronunciation period of a single word at time t of the audio information and the time of the two complete pronunciation periods of the single word before and after, N = 3f s / k,f s is the audio sampling frequency.

[0275] Specifically, N is the time required to pronounce the word in the lyrics corresponding to time t in the song, and the sum of the time required to pronounce the two previous and next words.

[0276] The amplitude adjustment module 90 is connected to the lip shape generation module and is used to adjust the amplitudes of several lip shapes according to the energy of the audio information.

[0277] Specifically, Fig.14 As shown, the amplitude adjustment module 90 includes:

[0278] The first average energy calculation unit 91 is used to calculate the first average energy of the audio information within M seconds, and the formula is as follows:

[0279]

[0280] where e t is the first average energy, t is the current time of the audio information, and the unit of t is seconds;

[0281] For example, j = 1, 2, 3 ... T, a song is usually 4 to 5 minutes long, and a verse or chorus is about 1 minute long, so a 1-minute window is selected to calculate the average energy, that is, M is equal to 60, and the average energy at time t is:

[0282]

[0283] The second average energy calculation unit 92 is connected to the first average energy calculation unit 91 and is used to calculate the second average energy after the modulation ratio is regularized. The formula is as follows:

[0284]

[0285] where r t is the second average energy, is the first average energy vector within 1 to T seconds.

[0286] The lip shape adjustment unit 93 is connected to the second average energy calculation unit 93, and is used to adjust the lip shape according to the second average energy.

[0287] Specifically, the formula for adjusting the lip shape is:

[0288]

[0289] in is the modulated lip shape parameter, l t It is the lip shape parameter before debugging.

[0290] In this embodiment, a system for synthesizing facial animation of virtual human singing is proposed, which fills the gap of complete facial synthesis algorithm for virtual human singing. In the system proposed in this embodiment, different modules synthesize and merge the two most important elements of the face, namely, mouth shape and expression, respectively, to generate rich, realistic and varied facial animation. This embodiment focuses on the application scenario of virtual human singing and proposes a complete solution. In this embodiment, the influence of different speech speeds on facial expression animation is considered when the virtual human sings with a wide range of speech speeds. The facial expression changes are generated according to the different speech speeds during the singing process, and the influence of each line of lyrics on the facial expression is comprehensively generated to generate the facial animation of the virtual human during the singing process. At the same time, nonlinear smoothing is used in the process of mouth shape changes, and the amplitude of the mouth shape of the virtual human during singing is modulated according to the volume, which can improve the matching degree of the mouth shape and the song, making the facial expression of the virtual human richer and more natural when singing.

[0291] Example 9

[0292] According to one embodiment of the present invention, there is provided a storage medium storing at least one instruction for implementing the operations performed by the facial animation generation method described in Embodiments 1 to 7.

[0293] They can be implemented with program codes executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0294] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0295] Those of ordinary skill in the art will appreciate that the units and steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed with hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0296] In the embodiments provided in the present application, it should be understood that the disclosed facial animation generation method, system and storage medium can be implemented in other ways. For example, the facial animation generation method, system and storage medium embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units or integrated circuits, which can be electrical, mechanical or other forms.

[0297] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0298] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0299] It should be noted that the above embodiments can be freely combined as needed. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention.

Claims

1. A method for generating facial animation, characterized in that: Includes steps: Receive audio and text messages; generating a plurality of lip shapes according to the audio information; Calculating the lip shape change rate of the audio information; Calculate the first average energy of the audio information within M seconds, the formula is as follows: where e t is the first average energy, t is the current time of the audio information, the unit of t is seconds, and f s is the audio sampling frequency; The modulation ratio is calculated as the second average energy after regularization, and the formula is as follows: where r t is the second average energy, is the first average energy vector within 1 to T seconds; Adjusting the lip shape according to the second average energy; Generate a lip shape animation according to the lip shape change rate and a plurality of lip shapes; Using a linear interpolation method to linearly smooth the lip animation; Define the Gaussian smoothing window, the formula is as follows: in, k is the lip shape change rate, i is the relative time scale of the audio information, and e is a constant; The lip animation is subjected to nonlinear smoothing processing, and the formula is as follows: in is the mouth shape at the current moment after the nonlinear smoothing process, N is the smoothing window width, N=3f s / k; Obtaining the global expression of the text information and the weights of several preset expressions of each part of the text information; Generate an expression animation according to the weights of the global expression and a plurality of preset expressions in each part of the text information; Facial animation is generated according to the lip animation and the facial expression animation.

2. The method for generating facial animation according to claim 1, characterized in that: The generating of a plurality of lip shapes according to the audio information specifically includes: Convert each part of the audio information into a number of phonemes; Generate a corresponding first mouth shape according to each of the phonemes; Respectively obtain a number of complete pronunciation time periods of each part of the audio information; Time-aligning each of the complete pronunciation periods with the corresponding phonemes; The first mouth shape corresponding to the phoneme in the middle position of each complete pronunciation period is identified as the key mouth shape.

3. The method for generating facial animation according to claim 2, characterized in that: The generating of the lip shape animation according to the lip shape change rate and a plurality of the lip shapes specifically includes: Generate a plurality of lip animations according to the lip shape change rate and a plurality of lip shapes corresponding to each part of the audio information; Combining several segments of the sub-lip animations in order to generate the lip animation; The step of generating a plurality of segment lip animations according to the lip shape change rate and the lip shapes corresponding to the respective parts of the audio information specifically includes: When the mouth shape change rate is less than a preset threshold, generating the sub-mouth shape animation according to a plurality of the first mouth shapes corresponding to each part of the audio information; When the lip shape change rate is greater than a preset threshold, the sub-lip shape animation is generated according to a plurality of the key lip shapes corresponding to each part of the audio information.

4. The method for generating facial animation according to claim 1, characterized in that: The obtaining of the global expression of the text information specifically includes: Calculating the weights of several preset emotions in the text information according to a preset emotion classification model, and respectively calculating the weights of several preset emotions in each part of the text information; Converting a plurality of the preset emotions into corresponding preset expressions according to a preset expression library; The preset expression with the highest weight in the text information is identified as the global expression.

5. The method for generating facial animation according to claim 4, characterized in that: The generating of expression animation according to the weights of the global expression and several preset expressions in each part of the text information specifically includes: Combining the global expression and a plurality of the preset expressions to generate a plurality of local expressions by using a weighted average method; A plurality of the local expressions are combined in sequence to generate the expression animation.

6. A facial animation generation system, characterized in that: include: A receiving module, used for receiving audio information and text information; A lip shape generation module, connected to the receiving module, for generating a plurality of lip shapes according to the audio information; A rate calculation module, connected to the receiving module, for calculating the rate of change of the outlet pattern according to the audio information; an amplitude adjustment module, connected to the lip shape generation module, for adjusting the amplitudes of a plurality of the lip shapes according to the energy of the audio information; The amplitude adjustment module includes a first average energy calculation unit, a second average energy calculation unit and a lip shape adjustment unit; The first average energy calculation unit calculates the first average energy of the audio information within M seconds, and the formula is as follows: where e t is the first average energy, t is the current time of the audio information, the unit of t is seconds, and f s is the audio sampling frequency; The second average energy calculation unit is used to calculate the second average energy after the modulation ratio is regularized, and the formula is as follows: where r t is the second average energy, is the first average energy vector within 1 to T seconds; The lip shape adjustment unit is used to adjust the lip shape according to the second average energy; A lip shape animation generation module, connected to the lip shape generation module and the rate calculation module, for generating a lip shape animation according to the lip shape change rate and a plurality of lip shapes; A smoothing module, connected to the lip animation generation module, for performing nonlinear smoothing processing on the lip animation; The smoothing module includes a linear smoothing unit, a smoothing window definition unit and a nonlinear smoothing unit. The linear smoothing unit is used to perform linear smoothing on the lip animation by using a linear interpolation method; The smoothing window definition unit is used to define a Gaussian smoothing window, and the formula is as follows: in, k is the lip shape change rate, i is the relative time scale of the audio information, and e is a constant; The nonlinear smoothing unit is used to perform nonlinear smoothing processing on the lip animation, and the formula is as follows: in is the mouth shape at the current moment after the nonlinear smoothing process, N is the smoothing window width, N=3f s / k; An acquisition module, connected to the receiving module, for acquiring the global expression of the text information and the weights of several preset expressions of various parts in the text information; An expression animation generation module, connected to the acquisition module, for generating an expression animation according to the weights of the global expression and a plurality of preset expressions in each part of the text information; A facial animation generation module is connected to the lip animation generation module and the facial expression animation generation module, and is used to generate facial animation according to the lip animation and the facial expression animation.

7. A storage medium, characterized in that: The storage medium stores at least one instruction, and the instruction is used to implement the operation performed by the facial animation generation method according to any one of claims 1 to 5.

Images