Voice mouth shape synchronization system based on phoneme and face posture mixing

Abstract

The invention discloses a voice mouth shape synchronization system based on mixing of phonemes and facial postures, relates to the field of computer animation and voice synthesis, realizes an efficient text-to-voice and mouth shape synchronization technology, and comprises the following steps: firstly, converting a text into voice and generating phoneme information through a TTS (Text To Speech) service, and then, according to the time length of the voice, carrying out voice and mouth shape synchronization; the time point of each phoneme peak value is accurately calculated, a designer prepares the mouth shape posture corresponding to each phoneme in advance, the weight of the previous phoneme is gradually reduced between the two phonemes, the weight of the next phoneme is increased, each phoneme corresponds to a curve in the unreal engine posture assets, and the unreal engine posture assets are obtained. The curve value is modified by dynamically adjusting the phoneme weight, so that the final mouth shape animation is generated, in addition, more vivid facial expressions are realized through the facial animation blueprint and the mixed emotion value, the scheme is excellent in real-time performance and network delay, only 20 animation sequence assets are needed, compared with a traditional scheme, resource consumption is reduced, and the method is suitable for popularization and application. And the efficiency is improved.

Description

Technical Field

[0001] The invention relates to the fields of computer animation and speech synthesis, and in particular to a speech lip-sync system based on a mixture of phonemes and facial gestures. Background Art

[0002] In modern multimedia technology, text-to-speech (TTS) systems have been widely used in various scenarios, such as virtual assistants, navigation systems, and educational software. However, in the existing technology, how to synchronize text-to-speech with lip sync in animation remains a challenge. Most existing solutions either require a lot of manual adjustments or have insufficient synchronization accuracy. Therefore, it is of great significance to propose a speech lip synchronization system based on a mixture of phonemes and facial gestures. Summary of the invention

[0003] The object of the present invention is to provide a speech lip synchronization system based on a mixture of phonemes and facial gestures, which realizes high-precision synchronization of text to speech and lip animation through two-stage processing.

[0004] To achieve the above object, the present invention provides the following technical solution: a speech lip synchronization system based on a mixture of phonemes and facial gestures, comprising the following steps:

[0005] Enter text:

[0006] Enter the text that needs to be converted into speech in the input field;

[0007] Convert text to speech:

[0008] The computer then automatically converts the output text into speech through the TTS service;

[0009] Generate phoneme information:

[0010] Split sentences into individual sentences according to punctuation marks, process each sentence separately, and use phoneme analysis tools to convert text into corresponding phonemes;

[0011] Use the difference algorithm to calculate the weight of the phoneme at each time point:

[0012] Calculate the time each phoneme affects based on the speech duration and the time proportion of each phoneme;

[0013] Generate lip animation based on weights:

[0014] Then the weight of the phonemes is calculated to match the corresponding speech lip shapes in the database;

[0015] Mixed sentiment values:

[0016] Then, the weight of the phoneme is fine-tuned by using the difference transition, so as to achieve the purpose of inserting the emotion value;

[0017] Play speech and lip animation:

[0018] Finally, connect the lip shapes to form a complete voice animation, and match the voice animations.

[0019] Optionally, a lip shape module, a phoneme module, a correspondence module and a matching module are used;

[0020] The mouth shape module is used to store 20 mouth shapes formed by emitting 48 phonemes;

[0021] The phoneme module is used to store 48 phonemes;

[0022] The corresponding module is used to match the mouth shapes corresponding to the pronunciation of each phoneme one by one;

[0023] The matching module is used to match the corresponding matching data between the phonemes in the corresponding module and the mouth shapes.

[0024] Optionally, the speech lip synchronization system mixed with phonemes and facial gestures further comprises a computing module;

[0025] The calculation module comprises:

[0026] A phoneme time proportion calculation unit and a difference transition calculation unit;

[0027] The phoneme time proportion calculation unit is used to calculate the factor weight data of each sentence processed separately for each sentence of the sentence split according to punctuation marks;

[0028] The difference transition calculation unit is used to fine-tune the calculation of the weight of the phoneme when mixing the emotion values.

[0029] Optionally, the phoneme time proportion calculation unit is as follows:

[0030]

[0031] in:

[0032] is the weight of the ith phoneme;

[0033] T is the total duration of speech;

[0034] is the time proportion of the i-th phoneme;

[0035] n is the total number of phonemes.

[0036] Optionally, the phoneme is divided into a highest point, a starting point and an ending point:

[0037] The peak time of each phoneme , the phoneme value is , according to itself and the following phonemes, it can be divided into 4 cases:

[0038] Vowel-vowel, vowel-consonant, consonant-vowel, consonant-consonant;

[0039] Configure 4 peak times:

[0040] The starting time of each phoneme , when the phoneme value is 0, it is the highest point of the previous phoneme. If it is the first phoneme, it is time 0;

[0041] The end time of each phoneme , when the phoneme value is 0, it is the highest point of the next phoneme. If it is the last phoneme, it is delayed by 0.5 seconds to the end of the sentence.

[0042] Optionally, the difference transition calculation unit is as follows:

[0043]

[0044] in:

[0045] The highest point time of each phoneme, the phoneme value ;

[0046] It is the starting point time of each phoneme. When the phoneme value is 0, it is the highest point of the previous phoneme.

[0047] It is the end point time of each phoneme. When the phoneme value is 0, it is the highest point of the next phoneme.

[0048] And when t is less than or greater than , the phoneme size is 0.

[0049] Optionally, the matching module includes:

[0050] An input unit and a search unit, the input text information is transported to the input unit, and then the input unit transmits the information to the search unit after identification. The search unit sequentially matches the data in the input unit with the phoneme elements in the phoneme table until a phoneme matching the information in the input unit is found.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] 1. The present invention first converts text into speech and generates phoneme information through TTS service, and then accurately calculates the time point of each phoneme peak according to the duration of the speech.

[0053] Second, the designer of the present invention will prepare the lip pose corresponding to each phoneme in advance. In order to achieve a natural transition effect, the weight of the previous phoneme is gradually reduced between two phonemes, while the weight of the next phoneme is increased. Each phoneme corresponds to a curve in the Unreal Engine Pose Asset, and the curve value is modified by dynamically adjusting the phoneme weight to generate the final lip animation. In addition, through the facial animation blueprint, the emotion value can be mixed to achieve more realistic facial expressions.

[0054] 3. The present invention has excellent performance in real-time and network latency, and only requires about 20 animation sequence assets, which greatly reduces resource consumption and improves efficiency compared to traditional solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 The system flow chart of the speech lip synchronization system based on the mixture of phonemes and facial gestures;

[0056] Figure 2 A table diagram of factors for the speech lip-sync system based on a mixture of phonemes and facial gestures;

[0057] Figure 3 This is a time diagram of the influence of each phoneme after being calculated by the phoneme analysis tool in the speech lip synchronization system based on the mixture of phonemes and facial gestures;

[0058] Figure 4 Infographic of weight changes in the speech lip-sync system based on a mixture of phonemes and facial gestures. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] For examples, see Figure 1 , Figure 2 , Figure 3 and Figure 4 , this implementation provides a speech lip synchronization system based on a mixture of phonemes and facial gestures, comprising the following steps:

[0061] Enter text:

[0062] Enter the text that needs to be converted into speech in the input field;

[0063] Convert text to speech:

[0064] The computer automatically converts the output text into speech;

[0065] Generate phoneme information:

[0066] Split sentences into individual sentences according to punctuation marks, process each sentence separately, and use phoneme analysis tools to convert text into corresponding phonemes;

[0067] The difference algorithm is used to calculate the weight of the phonemes at each time point: the time affected by each phoneme is calculated based on the speech duration and the time proportion of each phoneme;

[0068] Generate lip animation based on weights:

[0069] Use the calculated phoneme weights to match the corresponding speech lip shapes in the database;

[0070] Mixed sentiment values:

[0071] Then, the difference transition is used to fine-tune the weight of the phoneme, thereby achieving the purpose of inserting the emotion value;

[0072] Play speech and lip animation:

[0073] Finally, connect the lip shapes to form a complete voice animation, and match the voice animations;

[0074] Using lip shape module, phoneme module, correspondence module and matching module;

[0075] The lip shape module is used to store 20 lip shapes formed by issuing 48 phonemes, the phoneme module is used to store 48 phonemes, the corresponding module is used to match the lip shapes corresponding to the sound of each phoneme one by one, the matching module is used to match the corresponding matching data of the phonemes and lip shapes in the corresponding module, and the voice lip shape synchronization system mixed with phonemes and facial gestures also includes a calculation module;

[0076] The computing module includes:

[0077] A phoneme time proportion calculation unit and a difference transition calculation unit;

[0078] The phoneme time proportion calculation unit is used to calculate the factor weight data of each sentence processed separately for each sentence split by punctuation marks;

[0079] The difference transition calculation unit is used to fine-tune the calculation of the weight of the phoneme when mixing the emotion value;

[0080] The phoneme time ratio calculation unit is as follows:

[0081]

[0082] in: is the weight of the ith phoneme;

[0083] T is the total duration of speech;

[0084] is the time proportion of the i-th phoneme;

[0085] n is the total number of phonemes;

[0086] Phonemes are divided into the highest point, the starting point and the ending point:

[0087] The peak time of each phoneme , the phoneme value is , according to itself and the following phonemes, it can be divided into 4 situations: vowel-vowel, vowel-consonant, consonant-vowel, consonant-consonant;

[0088] Configure the time of 4 highest points;

[0089] The starting time of each phoneme , when the phoneme value is 0, it is the highest point of the previous phoneme. If it is the first phoneme, it is time 0;

[0090] The end time of each phoneme , when the phoneme value is 0, it is the highest point of the next phoneme. If it is the last phoneme, it is delayed by 0.5 seconds at the end of the sentence;

[0091] The difference transition calculation unit is as follows:

[0092]

[0093] in The highest point time of each phoneme, the phoneme value ;

[0094] It is the starting point time of each phoneme. When the phoneme value is 0, it is the highest point of the previous phoneme.

[0095] It is the end point time of each phoneme. When the phoneme value is 0, it is the highest point of the next phoneme.

[0096] And when t is less than or greater than , the phoneme size is 0.

[0097] More specifically, in this embodiment: first, the text information is input into the input text, and then the input text is automatically converted into voice information through the TTS service, and then the sentence is automatically split into single sentences according to punctuation marks by the phoneme time proportion calculation unit, and then each single sentence is processed separately, and the time and weight of each phoneme influence are calculated, and then the corresponding voice lip shape in the database is matched according to the calculated result, and then according to the scene, the time and weight of each phoneme influence are fine-tuned by the difference transition calculation unit, so as to achieve the purpose of mixed emotion value, and finally all the sentences are combined and played;

[0098] Through the TTS service, the text is first converted into speech and phoneme information is generated. Then, the time point of each phoneme peak is accurately calculated based on the duration of the speech.

[0099] The designer will prepare the lip poses for each phoneme in advance. In order to achieve a natural transition effect, the weight of the previous phoneme is gradually reduced between two phonemes, while the weight of the next phoneme is increased. Each phoneme corresponds to a curve in the Unreal Engine Pose Asset, and the curve value is modified by dynamically adjusting the phoneme weight to generate the final lip animation. In addition, through the facial animation blueprint, the emotion values ​​can be mixed to achieve more realistic facial expressions.

[0100] See also Figure 1 , Figure 2 , Figure 3 and Figure 4 In this embodiment, the matching module includes an input unit and a search unit. The input text information is transported to the input unit, and then the input unit transmits the information to the search unit after identification. The search unit sequentially matches the data in the input unit with the phoneme elements in the phoneme table until a phoneme matching the information in the input unit is found.

[0101] In this embodiment, the real-time performance and network latency are excellent, and only 20 animation sequence assets are required, which greatly reduces resource consumption and improves efficiency compared to traditional solutions, thereby making the matching speed of facial animation and phonemes faster.

[0102] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A speech lip synchronization system based on a mixture of phonemes and facial gestures, characterized in that include: The lip shape module, the phoneme module, the corresponding module and the matching module specifically include the following steps: Enter text: Enter the text that needs to be converted into speech in the input field; Convert text to speech: The computer automatically converts the output text into speech through the TTS service; Generate phoneme information: Split sentences into individual sentences according to punctuation marks, process each sentence separately, and use phoneme analysis tools to convert text into corresponding phonemes; Use the difference algorithm to calculate the weight of the phoneme at each time point: Calculate the time each phoneme affects based on the speech duration and the time proportion of each phoneme; Generate lip animation based on weights: Use the calculated phoneme weights to match the corresponding speech lip shapes in the database; Mixed sentiment values: The weight of the phoneme is fine-tuned by using the difference transition to achieve the purpose of inserting the emotion value; Play speech and lip animation: Connect the lip shapes to form a complete voice animation, and then match the voice animations.

2. A speech lip synchronization system based on a mixture of phonemes and facial gestures according to claim 1, characterized in that: The mouth shape module is used to store 20 mouth shapes formed by emitting 48 phonemes; The phoneme module is used to store 48 phonemes; The corresponding module is used to match the mouth shapes corresponding to the pronunciation of each phoneme one by one; The matching module is used to match the corresponding matching data between the phonemes in the corresponding module and the mouth shapes.

3. A speech lip synchronization system based on a mixture of phonemes and facial gestures according to claim 2, characterized in that: The voice lip synchronization system mixed with phonemes and facial gestures also includes a computing module; The calculation module comprises: A phoneme time proportion calculation unit and a difference transition calculation unit; The phoneme time proportion calculation unit is used to calculate the factor weight data of each sentence processed separately for each sentence of the sentence split according to punctuation marks; The difference transition calculation unit is used to fine-tune the calculation of the weight of the phoneme when mixing the emotion values.

4. A speech lip synchronization system based on a mixture of phonemes and facial gestures according to claim 3, characterized in that: The phoneme time proportion calculation unit is as follows: ; in: is the weight of the ith phoneme; T is the total duration of speech; is the time proportion of the i-th phoneme; n is the total number of phonemes.

5. A speech lip synchronization system based on a mixture of phonemes and facial gestures according to claim 4, characterized in that: The phonemes are divided into the highest point, the starting point and the ending point: The peak time of each phoneme , the phoneme value is , according to itself and the following phonemes, it can be divided into 4 cases: Vowel-vowel, vowel-consonant, consonant-vowel, consonant-consonant; Configure 4 peak times: The starting time of each phoneme , when the phoneme value is 0, it is the highest point of the previous phoneme. If it is the first phoneme, it is time 0; The end time of each phoneme , when the phoneme value is 0, it is the highest point of the next phoneme. If it is the last phoneme, it is delayed by 0.5 seconds to the end of the sentence.

6. A speech lip synchronization system based on a mixture of phonemes and facial gestures according to claim 3, characterized in that: The difference transition calculation unit is as follows: ； in The highest point time of each phoneme, the phoneme value ; It is the starting point time of each phoneme. When the phoneme value is 0, it is the highest point of the previous phoneme. It is the end point time of each phoneme. When the phoneme value is 0, it is the highest point of the next phoneme. And when t is less than or greater than , the phoneme size is 0.

7. A speech lip synchronization system based on a mixture of phonemes and facial gestures according to claim 2, characterized in that: The matching module includes: An input unit and a search unit, the input text information is transported to the input unit, and then the input unit transmits the information to the search unit after identification. The search unit sequentially matches the data in the input unit with the phoneme elements in the phoneme table until a phoneme matching the information in the input unit is found.

Images