Text-driven digital human high-precision sound and lip synchronization system and text-driven digital human high-precision sound and lip synchronization method

Abstract

The invention discloses a text-driven digital human high-precision sound and lip synchronization system and method, and the system comprises a text module which is used for converting an input text into linguistic features, including phonemes and rhythms; the acoustic module is used for converting the linguistic features into acoustic features and phoneme animation data; the audio module is used for sampling the acoustic features to generate audio clips; and the digital human interface module is used for receiving the phoneme animation data and the audio clips and synchronously playing the phoneme animation data and the audio clips so as to realize digital human lip synchronization. According to the system, high-precision audio and lip synchronization of the digital human is driven by a text, high-quality audio and lip animation data can be generated at the same time, the voice interaction fluency, the natural sense and the trueness of the digital human are greatly improved, and the user experience is improved. And meanwhile, the generation cost is reduced, and the generation efficiency is improved. Besides, a unified model of audio and phoneme animation data is generated from the text by using the neural network, so that synchronous learning of the sound spectrum and the animation data can be ensured, and the precision is higher.

Description

Technical Field

[0001] The present invention relates to the technical field of computer information processing, and more particularly to a text-driven digital human high-precision lip synchronization system and method. Background Art

[0002] Digital human refers to a virtual character that exists in digital form in digital space and has anthropomorphic or real-life appearance, behavior and characteristics, involving a variety of technologies such as modeling, physical simulation, rendering, motion capture, facial capture and artificial intelligence. With the continuous improvement of artificial intelligence technology and computer technology, the application and demand of digital human are growing continuously. It has been widely applied in many fields such as metaverse, live broadcast, variety show, game, finance, media, culture and tourism, science and education, pan-entertainment, health care, etc. The specific presentation forms include virtual hosts, virtual anchors, fashion show models, virtual idols, virtual teachers, short video virtual images, etc.

[0003] Digital humans can reduce repetitive human labor, stay online 24 hours a day, and greatly improve the production efficiency of the industry. High-quality digital humans are similar to real people and need to have both "form and spirit". "Form" refers to very realistic hair, skin and other visual effects, and "spirit" refers to vivid and flexible movements, expressions, and lip animation details.

[0004] With the advancement of rendering technology, the appearance of digital humans can achieve very realistic effects. For example, companies such as Unreal, Unity, and Adobe have launched their own digital human rendering systems, which can achieve next-generation high-precision rendering. However, many digital humans are only similar in appearance but not in spirit. They are not lively and flexible enough, and are relatively rigid. They do not have rich expressions, movements, and lip animations. Sometimes, lip animations and sounds are difficult to match accurately, which makes digital humans look like exquisite models without souls, reducing the experience of digital humans, greatly affecting the promotion of digital humans, and sometimes even bringing about the "uncanny valley effect."

[0005] Only with anthropomorphic thoughts and behaviors, and vivid and flexible animation details, can digital humans bring users a sense of intimacy, participation, interaction, and immersion. To be like a real person, digital humans must at least have natural and smooth body movements, natural facial expressions, and lip sync.

[0006] Related technologies:

[0007] 1. Random lip movement: This is the simplest lip movement solution. These methods randomly play lip movement animations to create a feeling of speaking, but the accuracy of this random playback is very low, which greatly reduces the look and feel of the digital human and its realism. It is only suitable for scenes with very rough models or some 2D images.

[0008] 2. Audio-driven lip movement technologies, such as Wav2lip and lipsync, require audio input, which is more expensive than text input. Due to the lack of precise phoneme information, common audio-driven lip movement methods are usually unable to accurately synthesize lip shapes when processing dynamic and unconstrained speaking situations, resulting in the generated lip animation being out of sync with the audio and having poor accuracy, which makes users feel unreal and disconnected.

[0009] 3. Video-driven technology, such as Apple's face capture technology. Face capture-based methods can achieve relatively accurate results in terms of accuracy, but there are two major disadvantages that make it difficult to apply on a large scale:

[0010] 1) Real people are required to record in front of professional equipment every time, which consumes a lot of manpower and is not suitable for fast and efficient deployment and daily use.

[0011] 2) The equipment is relatively expensive and requires better face capture hardware support. Ordinary users are not willing to buy it, and the operating cost is also relatively high.

[0012] Therefore, the above-mentioned defects limit the ability of digital human technology to enhance user participation, interaction and immersion, especially in scenarios that require high degree of simulation, and existing technical means are difficult to fully meet market demand. Summary of the invention

[0013] In view of this, the present invention provides a text-driven high-precision lip synchronization system and method for digital humans, which is dedicated to solving the lip synchronization problem of digital humans and improving the existing problems of inaccurate lip animation, unnatural audio, lack of details and lip synch.

[0014] In order to achieve the above object, the present invention adopts the following technical solution:

[0015] In a first aspect, the present invention provides a text-driven digital human high-precision lip synchronization system, comprising:

[0016] The text module converts the input text into linguistic features, including phonemes and prosody;

[0017] An acoustic module, used for converting the linguistic features into acoustic features and phoneme animation data;

[0018] An audio module, used for sampling the acoustic features to generate an audio clip;

[0019] The digital human interface module is used to receive the phoneme animation data and audio clips and play them synchronously to achieve digital human lip synchronization.

[0020] Furthermore, the text module includes:

[0021] a text normalization unit for converting input text as written text words into spoken words;

[0022] Generate a phoneme unit, which is used to implement polyphone disambiguation processing on the spoken word using a Bert model and a SoftMax classifier to generate corresponding phonemes;

[0023] The prosody prediction unit is used to perform prosody analysis on the spoken words based on the Bert model and the SoftMax classifier, and output a corresponding prosody level.

[0024] Furthermore, the acoustic module comprises:

[0025] A vectorization unit, used to convert the phoneme and prosody information generated by the text module into an embedded vector form through embedding technology;

[0026] The encoder unit uses a feed-forward Transformer block to analyze and understand phonemes and their associated prosodic features, extracting the contextual semantic associations of phonemes and prosodic information;

[0027] A speech transformer unit for extracting speech features associated with phonemes, including: pitch, energy and weight, temporal information, and spectral-level sequence;

[0028] The autoregressive decoder unit uses a sequentially connected Prenet, GRU module and a fully connected layer structure to generate the final Mel spectrum for audio synthesis based on the input spectrum level sequence.

[0029] Furthermore, the encoder unit is composed of 4 feedforward Transformer blocks, the input is a 304-dimensional embedding vector, and the position code is added, which is converted into a hidden state sequence through the internal network structure of the encoder; the Transformer structure includes a multi-head self-attention layer and a one-dimensional convolution to extract the contextual semantic association of phonemes and rhythms.

[0030] Furthermore, the position encoding is used to provide the model with the position information of each phoneme in the sequence, using fixed position encoding of sine and cosine functions:

[0031] For the 2ith dimension at position pos in the phoneme sequence, it is an even dimension:

[0032]

[0033] For the 2i+1th dimension at position pos in the phoneme sequence, it is an odd dimension:

[0034]

[0035] Among them, pos is the position index, i is the dimension index, and dmodel is the dimension of the model's hidden layer.

[0036] Furthermore, the voice converter unit comprises:

[0037] A feature extractor for extracting acoustic features from the hidden state sequence output by the encoder unit based on a pre-trained recurrent neural network model;

[0038] A pitch predictor, configured to obtain pitch information from the acoustic features based on a pre-trained pitch prediction model;

[0039] an energy predictor, for obtaining energy information from the acoustic features based on a pre-trained energy prediction model;

[0040] A weight predictor, for obtaining weight information from the acoustic features based on a pre-trained weight prediction model;

[0041] The autoregressive length predictor encodes and concatenates the predicted pitch, energy, weight information and the rhythm and prosody extracted by the encoder unit, inputs a pre-trained autoregressive length prediction model, outputs the time information of the phoneme, and the time information passes through a length adjuster to output a spectrum-level sequence.

[0042] Furthermore, the pitch prediction model, energy prediction model and weight prediction model have the same network model structure, each consisting of two convolutional layers and one fully connected layer, are modeled separately, and are optimized using mean square error loss;

[0043] The pitch prediction model uses continuous wavelet transform to decompose the continuous pitch sequence into pitch spectrograms, and uses the pitch spectrograms as training targets;

[0044] The energy prediction model calculates the L2 norm of the amplitude of each short-time Fourier transform frame as energy; the energy is used as a training target;

[0045] The weight prediction model directly uses the animation weight of the pronunciation corresponding to the phoneme as a training target.

[0046] Furthermore, the audio module specifically adopts a HiFi-GAN network to upsample the Mel spectrum as an acoustic feature to generate high-quality audio; the HiFi-GAN network is trained using adversarial loss, Mel spectrum loss and feature matching loss.

[0047] Furthermore, the digital human interface module comprises:

[0048] The lip animation interface layer is used to receive the phoneme weights and time information generated by the acoustic module and convert them into corresponding lip animations;

[0049] The audio playback interface layer is used to receive the audio generated by the audio module and play the audio synchronously with the lip animation.

[0050] In a second aspect, an embodiment of the present invention further provides a text-driven digital human high-precision lip synchronization method, using the text-driven digital human high-precision lip synchronization system as described in any one of the first aspects, comprising the following steps:

[0051] Take input text and convert it into linguistic features, including phonemes and prosody, through the text module;

[0052] The linguistic features output by the text module are converted into acoustic features and phoneme animation data through the acoustic module;

[0053] Using the audio module to sample acoustic features and generate corresponding audio clips;

[0054] The phoneme animation data and audio clips are synchronously input into the digital human interface module to achieve digital human lip synchronization.

[0055] It can be seen from the above technical solutions that compared with the prior art, the present invention has the following technical advantages:

[0056] 1. Both innovative and practical: The present invention innovatively proposes a text-driven high-precision lip synchronization system for digital humans, which can simultaneously generate high-quality audio and lip animation data, greatly improving the fluency, naturalness and authenticity of digital human voice interaction, and improving the existing problems of inaccurate lip animation, unnatural audio, lack of details and lip sync of digital humans, thereby improving the user experience.

[0057] 2. Most of the existing lip animations are designed based on English phonemes, which are difficult to perfectly cover Chinese scenes. The present invention is based on the "Hanyu Pinyin Scheme" standard and innovatively introduces the phonemes of Chinese initials and finals for learning and driving, so that the lip animation can cover 100% of the Chinese pronunciation scenes and achieve extremely high sound-lip accuracy in Chinese scenes.

[0058] 3. Easy to use, reducing the cost of generation and improving generation efficiency. The present invention does not require audio input, wearing complex facial capture equipment, etc., and only requires text input to generate high-precision lip synchronization. The system of the present invention will automatically generate audio data and animation data required for lip synchronization.

[0059] 4. Systematic learning framework: The present invention uses a neural network to generate a unified model of audio and phoneme animation data from text, which can ensure the synchronous learning of sound spectrum and animation data with higher accuracy; the pitch, energy and weight of the phoneme can provide better information supplement for the learning and prediction of the phoneme length, and obtain better time prediction information. All of these will make the final lip synchronization result more accurate.

[0060] 5. Repeatability and optimizability: The design of the present invention can continuously input new training data to strengthen the existing model and continuously optimize the effect; the modular design allows each subsequent module to be replaced with a more optimal component structure to further improve the effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0062] Figure 1 This is a structural diagram of the text-driven digital human high-precision lip synchronization system provided by the present invention.

[0063] Figure 2 A schematic diagram of text input and text module processing provided by the present invention.

[0064] Figure 3 A schematic diagram of phoneme generation provided by the present invention.

[0065] Figure 4 A schematic diagram of rhythm analysis provided by the present invention.

[0066] Figure 5 A schematic diagram of the acoustic module processing of phoneme and prosodic input provided by the present invention.

[0067] Figure 6 A processing schematic diagram of the voice converter provided by the present invention.

[0068] Figure 7 This is a schematic diagram of feature extractor processing in the speech converter provided by the present invention.

[0069] Figure 8 This is a structural diagram of the pitch, energy and weight predictor in the speech converter provided by the present invention.

[0070] Fig. 9 This is a structural diagram of the autoregressive length predictor in the speech converter provided by the present invention.

[0071] Fig.10 This is a schematic diagram of the encoder processing provided by the present invention.

[0072] Fig.11 A schematic diagram of the structure of the autoregressive decoder in the acoustic module provided by the present invention.

[0073] Fig.12 This is a processing schematic diagram of the audio module provided by the present invention. DETAILED DESCRIPTION

[0074] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0075] Most of the existing lip animations are designed based on English phonemes, which makes it difficult to perfectly cover Chinese scenes. However, the present invention uses the initials, finals and tones in the "Chinese Pinyin Scheme" issued by the Ministry of Education as basic phonemes, and trains the neural network model to output anthropomorphic audio and the lip animation data (phonemes, time and weight) that accurately corresponds to it. The audio and lip animation data are respectively input into the interface modules corresponding to the digital human, and finally the high-precision audio-lip synchronization of the digital human is achieved.

[0076] The present invention innovatively introduces the phonemes of Chinese initials and finals for learning and driving, so that the lip shape animation covers 100% of the Chinese pronunciation scenes and achieves extremely high sound-lip accuracy in Chinese scenes.

[0077] Embodiment 1:

[0078] The embodiment of the present invention discloses a text-driven digital human high-precision lip synchronization system, which includes four parts: a text module, an acoustic module, an audio module and a digital human interface module. The overall framework is as follows: Figure 1 As shown:

[0079] The text module converts the input text into linguistic features, including phonemes and prosody;

[0080] An acoustic module, used for converting the linguistic features into acoustic features and phoneme animation data;

[0081] An audio module, used for sampling the acoustic features to generate an audio clip;

[0082] The digital human interface module is used to receive the phoneme animation data and the audio clip and play them synchronously to achieve digital human lip synchronization.

[0083] The system inputs text through the text module, which converts the input text into linguistic features (phonemes and rhythm). The acoustic module further converts the linguistic features into acoustic features and phoneme animation data. The audio module is responsible for sampling the acoustic features to generate audio clips. The phoneme animation data and audio clips are synchronously input into the corresponding lip animation interface and audio playback interface of the digital human, so that high-precision audio playback and lip animation synchronization can be achieved.

[0084] The technical solution of the present invention is described in more detail below by way of example:

[0085] For example, input the corresponding plain text for the digital human to speak, such as "I am a digital human".

[0086] 1. Refer to Figure 2 As shown, the text input text module extracts valid text information. There are the following problems from text to Chinese pronunciation: the conversion from written language to spoken language, intonation, and polyphonic characters. These problems are solved through the text module. The text module contains three functional units: a text normalization unit, a phoneme generation unit, and a prosody prediction unit.

[0087] 1) The text normalization unit is responsible for converting written text words such as numbers, symbols, and abbreviations into spoken words, and converting the input text into a unified standard form. For example, "23.56 kg" is normalized to "twenty-three point five six kilograms". This is achieved through the method of regular expression + rule matching.

[0088] 2) The phoneme generation unit. The problem of polyphonic characters widely exists in Chinese. For example, "digital" and "counting", polyphonic characters are related to the surrounding words and the overall context. One of the key points in generating phonemes is to disambiguate polyphonic characters. For Chinese characters with fixed pronunciations, they can be directly stored in the dictionary and the method of dictionary query is used to generate phonemes; for texts involving polyphonic characters, a pre-trained Bert model + SoftMax classifier is used to achieve polyphonic character disambiguation. As Figure 3 shown, the output result is "wo3 shi4 shu4 zi4 ren2", and the pronunciations of the 5 characters are the 3rd tone, 4th tone, 4th tone, 4th tone, and 2nd tone respectively. The Bert model is "bidirectional encoding representation based on transformer", which has context semantic analysis and self-attention mechanism. These features can capture complex dependencies in the input sequence and improve the recognition accuracy of polyphonic characters.

[0089] Among them, the Bert model: The Bert model is pre-trained on a large-scale corpus so that it can master rich context semantic information. The pre-training tasks include masked language model and next sentence prediction. These tasks help Bert learn the semantic relationships between words and sentence structures. In specific applications, such as the polyphonic character disambiguation task, the pre-trained Bert model is usually fine-tuned. The fine-tuning process is to further train the model on a task-related dataset to make it more accurate in this task.

[0090] SoftMax classifier: Based on the Bert model, a SoftMax layer is added to map the representation vectors generated by Bert to specific classification labels (i.e., different pronunciations). In the polyphonic character disambiguation task, each possible pronunciation can be regarded as a classification label.

[0091] The application process is as follows:

[0092] When used, text input is first processed by a pre-trained and fine-tuned Bert model to generate context-dependent semantic vectors. These semantic vectors are then fed into a SoftMax classifier, which predicts the most likely pronunciation based on contextual information, thereby achieving disambiguation of polyphones.

[0093] 3) Prosody prediction unit. Prosody includes information such as intonation. Correct prosody analysis can help generate more realistic speech. This embodiment uses four prosody levels to represent the intonation in Chinese pronunciation: L0 (phoneme level), L1 (word), L3 (breathing pause), and L4 (end of sentence). This embodiment also uses a pre-trained Bert model + SoftMax classifier to implement prosody analysis, such as Figure 4 As shown, the output result is the prosody levels of 5 words “L1 L1 L0 L0 L3”.

[0094] For the prosody analysis task, the pre-trained Bert model is fine-tuned on a specific prosody annotation dataset. Through fine-tuning, the Bert model can better understand the text features related to prosody and improve the prediction accuracy of prosody level.

[0095] SoftMax classifier: Receives the vector representations generated by the Bert model and maps these vectors to different prosody levels (e.g. L0, L1, L3, L4, etc.). In the prosody analysis task, each possible prosody level is considered as a classification label.

[0096] The application process is as follows:

[0097] The input text sequence first passes through the pre-trained and fine-tuned Bert model to generate context-related semantic vectors. These vectors are input to the SoftMax classifier, which outputs the corresponding prosody level based on the contextual information of the text. In this way, the Bert model + SoftMax classifier can effectively analyze the prosody information of the text and help generate more natural and realistic speech.

[0098] 2. The phonemes and rhythms output by the text module are further processed by the acoustic module, such as Figure 5 shown.

[0099] The Acoustics Module consists of four functional units:

[0100] A vectorization unit, used to convert the phoneme and prosody information generated by the text module into an embedded vector form through embedding technology;

[0101] The encoder unit uses a feed-forward Transformer block to analyze and understand phonemes and their associated prosodic features, extracting the contextual semantic associations of phonemes and prosodic information;

[0102] A speech transformer unit for extracting speech features associated with phonemes, including: pitch, energy and weight, temporal information, and spectral-level sequence;

[0103] The autoregressive decoder unit uses a sequentially connected Prenet, GRU module and a fully connected layer structure to generate the final Mel spectrum for audio synthesis based on the input spectrum level sequence.

[0104] First, the phonemes and rhythms are transformed into embedded vector representations through embedding technology. The original input of the phoneme is a one-hot 2208-dimensional vector, which is reduced to a 300-dimensional phoneme vector through the embedding matrix. The phoneme vector can better express the pronunciation relationship between phonemes. Rhythm embedding directly uses the one-hot technology to represent the four rhythm levels with a 4-dimensional rhythm vector. The phoneme vector and the rhythm vector are spliced ​​together and input into the encoder to convert from an embedded sequence to a hidden state sequence. The hidden state sequence enters the speech converter, and a part of the output is the phoneme animation data (phoneme weight + time) for lip animation playback; the other part of the output continues to be input into the autoregressive decoder for outputting the Mel spectrum.

[0105] 2.1. First, let us introduce the above-mentioned voice converter unit:

[0106] Voice converter unit, see Figure 6 ,include:

[0107] A feature extractor for extracting acoustic features from the hidden state sequence output by the encoder unit based on a pre-trained recurrent neural network model;

[0108] A pitch predictor, configured to obtain pitch information from the acoustic features based on a pre-trained pitch prediction model;

[0109] an energy predictor, for obtaining energy information from the acoustic features based on a pre-trained energy prediction model;

[0110] A weight predictor, for obtaining weight information from the acoustic features based on a pre-trained weight prediction model;

[0111] The autoregressive length predictor encodes and concatenates the predicted pitch, energy, weight information and the rhythm and prosody extracted by the encoder unit, inputs a pre-trained autoregressive length prediction model, outputs the time information of the phoneme, and the time information passes through a length adjuster to output a spectrum-level sequence.

[0112] The speech converter adds duration, pitch, energy, and weight information to the hidden state sequence, allowing the model to better grasp the audio features. Duration affects the length of pronunciation; pitch affects emotion and rhythm; energy affects the amplitude of the spectrum, which directly affects the volume of the audio; weight information is a 47-dimensional vector, corresponding to initials and finals, and affects lip animation. Its internal details are as follows Figure 6 The processing process:

[0113] The hidden state sequence is input into the feature extractor to obtain features, and the features are further input into three predictors: pitch, energy, and weight predictors. The predicted pitch, energy, weight, phoneme, and prosody codes are concatenated and input into the autoregressive length predictor to obtain the time information of the phoneme, which is used to ensure the accurate correspondence between the lip animation and the audio timing. The time information is expanded into a spectral level sequence with the same dimension as the Mel spectrum through the length regulator (LR). The spectral level sequence is subsequently input into the autoregressive decoder to finally obtain the Mel spectrum.

[0114] 1) The feature extractor consists of a bidirectional long short-term memory network (BiLSTM), layer normalization (LN), a dropout layer, and a full connection layer (Full Connection), such as Figure 7 As shown in the figure. BiLSTM is a special recurrent neural network that can better capture the contextual information in the sequence by processing the input sequence in two directions; layer normalization can help the neural network converge faster and more stably; the dropout layer prevents overfitting and improves the generalization ability of the model; the fully connected layer is finally used to integrate the extracted features.

[0115] 2). Tone, energy and weight predictors have similar structures, such as Figure 8 As shown, it mainly consists of 2 convolutional layers and 1 fully connected layer. The three are modeled separately. The pitch predictor uses continuous wavelet transform to decompose the continuous pitch sequence into a pitch spectrogram, and uses the pitch spectrogram as the training target; the energy predictor calculates the L2 norm of the amplitude of each short-time Fourier transform (SIFT) frame as the energy; the weight predictor directly uses the animation weight of the pronunciation corresponding to the phoneme as the training target. All three predictors are optimized using mean square error (MSE) loss. The MSE calculation formula is as follows:

[0116]

[0117] Where N represents the total number of samples, y true，i Represents the true value of the i-th sample. pred，i Represents the prediction result of the i-th sample.

[0118] 3). The autoregressive length predictor includes Pre-net, LSTM and fully connected layers, such as Fig. 9 As shown in the figure. Pre-net performs a series of nonlinear transformations on the input, which helps the model converge and generalize. Pre-net includes two fully connected layers and uses the Relu activation function. Drop is 0.5 to improve generalization ability.

[0119] 2.2. The specific details of the specific encoder unit can adopt the following structure:

[0120] The encoder consists of 4 feed-forward Transformer blocks, such as Fig.10 As shown in the figure, the input is a 304-dimensional embedding vector, and the position code is added, which is converted into a hidden state sequence through the internal network structure of the encoder. The Transformer structure includes a multi-head self-attention layer (Multi-Head Attention) and a one-dimensional convolution, which can better extract the contextual semantic association of phonemes and rhythms.

[0121] Among them, the position coding is used to provide the model with the position information of each phoneme in the sequence. This embodiment adopts a coding mechanism similar to Transformer, using fixed position coding of sine and cosine functions. The specific method is:

[0122] For the 2ith dimension (even dimension) at position pos of the phoneme sequence:

[0123]

[0124] For the 2i+1th dimension (odd dimension) at position pos in the phoneme sequence:

[0125]

[0126] Where pos is the position index, i is the dimension index, and d model is the dimension of the hidden layer of the model. These functions ensure that the encodings at different positions are unique and smooth, which can maintain the stability and trainability of the model. The position encoding will be added to the embedding vector X to get the total input of the encoder X F :

[0127] X F =X+PE

[0128] 2.3. The structure of the autoregressive decoder is as follows Fig.11 As shown, it includes Prenet, GRU modules and fully connected layers.

[0129] 3. Audio module:

[0130] Reference Fig.12As shown in the figure, the Mel spectrum output by the acoustic module is continuously input into the audio module to generate the corresponding audio. The audio module uses the pre-trained HiFi-GAN network to upsample the Mel spectrum to generate high-quality audio. HiFi-GAN belongs to the generative adversarial network and is trained through three loss functions: the original adversarial loss GAN Loss, the Mel spectrum loss (converting the generated audio back to the Mel spectrum and calculating the L1 distance between the real and generated spectrums), and the feature matching loss. Finally, the trained HiFi-GAN can output high-quality audio by inputting the Mel spectrum.

[0131] 4. Digital human interface module:

[0132] like Figure 1 As shown, it includes: a lip animation interface layer, which is used to receive the phoneme weights and time information generated by the acoustic module and convert them into corresponding lip animations;

[0133] The audio playback interface layer is used to receive the audio generated by the audio module and play the audio synchronously with the lip animation.

[0134] The phoneme weights and time information obtained by the acoustic module will be input into the lip animation interface layer, and the audio generated by the audio module will be input into the digital human's audio playback interface layer to play the lip animation and audio synchronously, thereby achieving the effect of digital human lip synchronization.

[0135] The present invention uses a dual method of quantification and real-person evaluation to detect the final effect. The text of daily conversations can be randomly selected and input into the system, and the audio played by the digital human and the digital human lip animation can be recorded. Then the same text is pronounced in Mandarin by the real person, and the pronunciation process of the real person is recorded in video.

[0136] In the quantification scheme, for example, 18 feature points such as the corners of the mouth and the chin are marked around the lips, and the landmark distance between the digital human and the real person in the natural state and the corresponding speech state is calculated to measure the accuracy of the digital human's lip animation. The lip synchronization rate can reach 95%.

[0137] The specific calculation process is: record the positions of 18 points in the natural state respectively, the nose position PB i As a benchmark, for a certain lip animation, the position PC of 18 points of the digital human is calculated. i , and calculate the vector difference V with the natural state diff

[0138] V diff =PC i -PB i

[0139] The same method is used for real people speaking, and the vector difference V between real people animation and natural state is obtained rdiff

[0140] V rdiff =PC i -PB i

[0141] Finally, the average cosine similarity of the vector differences of the 18 points is calculated to measure the accuracy ACC

[0142]

[0143] Where dot represents the dot product operation of the vector, ||V diff || represents vector V diff The modulus (length).

[0144] In addition, a real-person questionnaire survey can be conducted to randomly play the digital human speech clips based on the present invention, and inquire about the naturalness of the voice, the accuracy of the lip shape animation, and whether the voice and lips are synchronized. The results show that 97% of the digital human clips are considered to have achieved the same lip synchronization effect as real people.

[0145] Both quantitative data indicators and real-person evaluations have proven that the present invention has achieved a high-precision lip synchronization solution.

[0146] Embodiment 2:

[0147] Based on the same inventive concept, an embodiment of the present invention also provides a text-driven digital human high-precision lip synchronization method. Since the principle of solving the problem by this method is similar to that of the aforementioned text-driven digital human high-precision lip synchronization system, the implementation of this method can refer to the specific implementation of the aforementioned system, and the repeated parts will not be repeated.

[0148] Using the text-driven digital human high-precision lip synchronization system as described in Example 1, the following steps are included:

[0149] (1) Obtain input text and convert it into linguistic features, including phonemes and prosody, through the text module;

[0150] (2) The linguistic features output by the text module are converted into acoustic features and phoneme animation data through the acoustic module;

[0151] (3) Using the audio module to sample acoustic features and generate corresponding audio clips;

[0152] (4) The phoneme animation data and audio clips are synchronously input into the digital human interface module to achieve digital human lip synchronization.

[0153] This method can achieve high-precision lip synchronization. By combining the text module, acoustic module and audio module, it can achieve accurate phoneme and rhythm analysis, ensuring that the generated audio is highly synchronized with the lip movements of the digital human. It solves the common problem of audio and lip animation being out of sync in the prior art and improves the fidelity of the digital human. It can quickly process input text and generate corresponding audio and lip animation, achieve real-time response, and enhance the user interaction experience.

[0154] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0155] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A text-driven digital human high-precision lip synchronization system, characterized in that: include: The text module converts the input text into linguistic features, including phonemes and prosody; An acoustic module, used for converting the linguistic features into acoustic features and phoneme animation data; An audio module, used for sampling the acoustic features to generate an audio clip; The digital human interface module is used to receive the phoneme animation data and the audio clip and play them synchronously to achieve digital human lip synchronization.

2. A text-driven digital human high-precision lip synchronization system according to claim 1, characterized in that: The text module comprises: a text normalization unit for converting input text as written text words into spoken words; Generate a phoneme unit, which is used to implement polyphone disambiguation processing on the spoken word using a Bert model and a SoftMax classifier to generate corresponding phonemes; The prosody prediction unit is used to perform prosody analysis on the spoken words based on the Bert model and the SoftMax classifier, and output a corresponding prosody level.

3. A text-driven digital human high-precision lip synchronization system according to claim 1, characterized in that: The acoustic module comprises: A vectorization unit, used to convert the phoneme and prosody information generated by the text module into an embedded vector form through embedding technology; The encoder unit uses a feed-forward Transformer block to analyze and understand phonemes and their associated prosodic features, extracting the contextual semantic associations of phonemes and prosodic information; A speech transformer unit for extracting speech features associated with phonemes, including: pitch, energy and weight, temporal information, and spectral level sequence; The autoregressive decoder unit uses a sequentially connected Prenet, GRU module and a fully connected layer structure to generate the final Mel spectrum for audio synthesis based on the input spectrum level sequence.

4. A text-driven digital human high-precision lip synchronization system according to claim 3, characterized in that: The encoder unit is composed of 4 feedforward Transformer blocks. The input is a 304-dimensional embedding vector, and the position code is added. It is converted into a hidden state sequence through the internal network structure of the encoder. The Transformer structure includes a multi-head self-attention layer and a one-dimensional convolution to extract the contextual semantic association of phonemes and rhythms.

5. A text-driven digital human high-precision lip synchronization system according to claim 4, characterized in that: The position encoding is used to provide the model with the position information of each phoneme in the sequence, using fixed position encoding of sine and cosine functions: For the 2ith dimension at position pos in the phoneme sequence, it is an even dimension: For the 2i+1th dimension at position pos in the phoneme sequence, it is an odd dimension: Among them, pos is the position index, i is the dimension index, and d model is the dimension of the model's hidden layer.

6. A text-driven digital human high-precision lip synchronization system according to claim 4, characterized in that: The voice converter unit comprises: A feature extractor for extracting acoustic features from the hidden state sequence output by the encoder unit based on a pre-trained recurrent neural network model; A pitch predictor, configured to obtain pitch information from the acoustic features based on a pre-trained pitch prediction model; an energy predictor, for obtaining energy information from the acoustic features based on a pre-trained energy prediction model; A weight predictor, for obtaining weight information from the acoustic features based on a pre-trained weight prediction model; The autoregressive length predictor encodes and concatenates the predicted pitch, energy, weight information and the rhythm and prosody extracted by the encoder unit, inputs a pre-trained autoregressive length prediction model, outputs the time information of the phoneme, and the time information passes through a length adjuster to output a spectrum-level sequence.

7. A text-driven digital human high-precision lip synchronization system according to claim 6, characterized in that: The pitch prediction model, energy prediction model and weight prediction model have the same network model structure, each consisting of two convolutional layers and one fully connected layer, and are modeled separately and optimized using mean square error loss. The pitch prediction model uses continuous wavelet transform to decompose the continuous pitch sequence into pitch spectrograms, and uses the pitch spectrograms as training targets; The energy prediction model calculates the L2 norm of the amplitude of each short-time Fourier transform frame as energy; the energy is used as a training target; The weight prediction model directly uses the animation weight of the pronunciation corresponding to the phoneme as a training target.

8. The text-driven digital human high-precision lip synchronization system according to claim 1, characterized in that: The audio module specifically adopts a HiFi-GAN network to upsample the Mel spectrum as an acoustic feature to generate high-quality audio; the HiFi-GAN network is trained using adversarial loss, Mel spectrum loss, and feature matching loss.

9. The text-driven digital human high-precision lip synchronization system according to claim 3, characterized in that: The digital human interface module comprises: The lip animation interface layer is used to receive the phoneme weights and time information generated by the acoustic module and convert them into corresponding lip animations; The audio playback interface layer is used to receive the audio generated by the audio module and play the audio synchronously with the lip animation.

10. A text-driven digital human high-precision lip synchronization method, characterized in that: The text-driven digital human high-precision lip synchronization system according to any one of claims 1 to 9 comprises the following steps: Take the input text and convert it into linguistic features, including phonemes and prosody, through the text module; The linguistic features output by the text module are converted into acoustic features and phoneme animation data through the acoustic module; Using the audio module to sample acoustic features and generate corresponding audio clips; The phoneme animation data and audio clips are synchronously input into the digital human interface module to achieve digital human lip synchronization.

Images