Digital human lip shape synchronization method based on voice driving

Abstract

The invention provides a digital human lip shape synchronization method based on voice driving. The method comprises the following steps: acquiring a voice signal; voice features related to lip shape changes in the voice signals are extracted; mapping the extracted voice features into lip shape change data; and applying the lip shape change data to the digital human model and carrying out real-time rendering to finish lip shape synchronization. According to the method, the voice signal characteristics are directly analyzed, subtle changes of pronunciation can be more accurately captured, and more natural and smooth lip-shaped actions are generated; the method is suitable for a multi-language environment, and especially improves the processing effect of tone languages such as Chinese and the like. An optimized signal processing and mapping algorithm improves the lip shape synchronization speed, and the real-time interaction requirement is met; a complicated phoneme identification process is not needed, and the technical implementation difficulty and the maintenance cost are reduced; through more accurate and natural lip shape synchronization, the overall performance and interaction naturalness of the digital human are remarkably improved.

Description

Technical Field

[0001] The invention relates to the field of human-computer interaction, and in particular to a digital human lip synchronization method based on voice drive. Background Art

[0002] With the rapid development of digital human technology and the continuous expansion of its application scope, people have put forward higher requirements for the realism and natural interaction of digital humans. Among the many factors that affect the realism of digital humans, lip synchronization technology is a crucial link. At present, the commonly used digital human lip synchronization method is mainly based on phoneme recognition, which converts the speech signal into a phoneme sequence and then maps it to the corresponding mouth shape movement. The above schemes have problems such as strong language dependence, limited accuracy, and low real-time performance. Especially when dealing with tonal languages ​​such as Chinese, the above schemes are difficult to accurately capture subtle changes in pronunciation, resulting in the generated lip shape movements not being natural and smooth enough. Summary of the invention

[0003] In view of the problems existing in the prior art, a digital human lip synchronization method based on voice drive is provided, which can complete the digital human lip synchronization by utilizing various features based on voice signals.

[0004] The technical solution adopted by the present invention is as follows: a digital human lip synchronization method based on voice drive, comprising: Acquire speech signals; Extract speech features related to lip shape changes in speech signals; Mapping the extracted speech features into lip shape change data; The lip shape change data is applied to the digital human model and rendered in real time to achieve lip synchronization.

[0005] As a preferred solution, the extracting speech features related to lip shape changes in the speech signal specifically includes: Extract syllable information, pitch information and sound length information from speech signals and integrate them into a comprehensive speech feature representation.

[0006] As a preferred solution, the syllable information extraction method includes: Calculate the short-time energy and zero-crossing rate of speech information, and set the judgment threshold of syllable boundary; The beginning and end of each syllable are identified according to the judgment threshold to obtain the syllable information in the speech signal.

[0007] As a preferred embodiment, the pitch information extraction method comprises: Perform autocorrelation analysis on speech signals to complete preliminary pitch estimation; Use cepstrum analysis to extract accurate fundamental frequency; The pitch contour is smoothed by median filtering to obtain a smoothed pitch change curve, i.e., pitch information.

[0008] As a preferred solution, the tone length information extraction method includes: Forced alignment of speech at the phoneme level; Calculate the exact duration of each phoneme, i.e. the sound length information.

[0009] As a preferred solution, linear prediction coding and cepstrum analysis are combined to complete the resonance peak analysis.

[0010] As a preferred solution, mapping the extracted speech features to lip shape change data specifically includes: Directly convert the extracted speech features into lip shape change data, or use a deep learning model to map the speech features into lip shape change data.

[0011] As a preferred solution, directly converting the extracted speech features into lip shape change data specifically includes: Based on the knowledge of phonetics and physiology, speech features are mapped into lip shape change data, where each syllable corresponds to a set of basic lip shape changes; pitch maps the vertical opening and closing degree of the lip shape; sound length maps the duration of the lip shape change; resonance peaks map the opening and closing degree and the degree of lip rounding of the lip shape; transition lip shapes are generated between adjacent phonemes; co-articulation effects are used to map adjacent phonemes; and speech rate features characterize the rate of lip shape change.

[0012] As a preferred solution, the method of using a deep learning model to map speech features into lip shape change data specifically includes: Establish a deep learning model based on convolutional neural network and long short-term memory network; wherein the deep learning model includes a connection input layer, a CNN module, an LSTM module, a fully connected layer and an output layer; the CNN module includes multiple convolutional layers and pooling layers, and the convolutional layer uses a multi-scale convolution kernel to capture speech features of different time spans; a residual connection is added between the CNN module and the LSTM module; and a self-attention mechanism is introduced in the LSTM module; Use adversarial training mechanism to complete the training of deep learning models; The trained model is used to map speech features to lip shape change data.

[0013] As a preferred solution, the method further includes interpolating the lip shape change data before applying the lip shape change data to the digital human model.

[0014] Compared with the prior art, the beneficial effects of adopting the above technical solution are: (1) High accuracy and naturalness: Direct analysis of speech signal features can more accurately capture subtle changes in pronunciation and generate more natural and smooth lip movements; (2) Strong language adaptability: It is suitable for a variety of language environments, and has particularly improved the processing effect of tonal languages ​​such as Chinese; (3) Good real-time performance: Optimized signal processing and mapping algorithms improve the speed of lip synchronization to meet the needs of real-time interaction; (4) Simple implementation: No complex phoneme recognition process is required, which reduces the difficulty of technical implementation and maintenance costs; (5) Enhanced user experience: Through more accurate and natural lip synchronization, the overall performance and natural interaction of digital humans are significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a flow chart of the voice-driven digital human lip synchronization method proposed by the present invention.

[0016] Figure 2 This is a flow chart of speech feature extraction proposed in one embodiment of the present invention. DETAILED DESCRIPTION

[0017] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar modules or modules with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be construed as limitations on the present application. On the contrary, the embodiments of the present application include all changes, modifications and equivalents that fall within the spirit and connotation of the appended claims.

[0018] The present invention mainly solves the following problems existing in the current digital human lip synchronization technology: (1) Lack of accuracy and naturalness: Especially when dealing with tonal languages ​​such as Chinese, it is difficult to accurately capture subtle changes in pronunciation; (2) Strong language dependence: Existing methods are mainly based on phoneme recognition and are difficult to adapt to different language environments; (3) Low real-time performance: Traditional methods have delays in processing and rendering, which affects the interactive experience; (4) Complex implementation: The phoneme recognition-based method requires a complex speech processing process, which increases the difficulty of implementation and maintenance costs.

[0019] Please refer to Figure 1 The embodiment of the present invention proposes a digital human lip synchronization method based on voice drive, which mainly realizes the synchronization of digital human lip shape based on voice signal processing, deep learning model, computer graphics technology, etc. The specific scheme is as follows: Step 1: Get the speech signal.

[0020] In this embodiment, the real-time received or pre-recorded voice information is mainly used as input for subsequent processing. The real-time received voice mailbox can be the voice collected in real time by a microphone, and the pre-recorded voice information can be a pre-recorded audio file.

[0021] After acquiring the speech signal, in order to ensure the accuracy of the speech signal, the speech signal is preprocessed by noise reduction and framing in this embodiment. Noise reduction can eliminate background noise and improve the clarity of the speech signal; framing divides the continuous speech signal into short time segments for subsequent analysis.

[0022] Step 2: Extract speech features related to lip shape changes in speech signals.

[0023] In this embodiment, please refer to Figure 2 , mainly extracts syllable information, pitch information and sound length information from speech signals, which can effectively characterize lip shape changes.

[0024] For syllable information, in this embodiment, the speech signal can be divided into syllables one by one, and the method combining short-time energy and zero-crossing rate is mainly used to identify and segment the syllable units in the speech signal. First, the short-time energy and zero-crossing rate of the speech signal are calculated.

[0025] The short-time energy calculation process is as follows: (1) Divide the speech signal into small time windows (usually 10-30ms); (2) Square the samples in each window; (3) Calculate the sum of the squares of the samples in each window; The calculation formula is: ,in, is a voice signal, is the window function.

[0026] The zero-crossing rate calculation process is as follows: (1) Divide the speech signal into small time windows (usually 10-30ms); (2) Count the number of times the signal changes from positive to negative or from negative to positive in each window; (3) Divide by the window length to get the zero-crossing rate; The calculation formula is: ,in, is a sign function, is the window length.

[0027] Since short-time energy and zero-crossing rate are used to characterize the energy distribution and frequency characteristics of speech signals, the syllable segmentation threshold can be set according to their calculation results. The specific method for obtaining the syllable separation threshold is as follows: (1) Calculate the average short-time energy of the entire speech ( E_avg ) and the average zero-crossing rate ( ZCR_avg ); (2) Setting energy threshold ( E_threshold ) is some multiple of the average energy: ,in, is the energy threshold coefficient, and in this embodiment, the value range is 1.1~1.5 (such as 1.2).

[0028] (3) Setting the zero-crossing rate threshold ( ZCR_threshold ) is some multiple of the average zero-crossing rate: Wherein, β is the zero-crossing rate threshold coefficient. In this embodiment, the value range is 1.2~2.0 (such as 1.5) The final syllable boundary judgment threshold is given by E_threshold and ZCR_threshold Together, they constitute the syllable boundary judgment threshold .

[0029] Based on the obtained syllable boundary judgment threshold, the syllable start and end judgment can be completed. Specifically, the syllable starts: when the short-time energy And the zero crossing rate ; Syllable ends: when short-term energy or zero crossing rate .

[0030] For high pitch information, in this embodiment, the autocorrelation method and the cepstrum method are mainly used to detect the fundamental frequency change of the voice to reflect the pitch change of the speaker. The specific extraction process is as follows: (1) Perform preliminary pitch estimation on the speech signal autocorrelation analysis. The specific calculation formula is as follows:

[0031] in, is the speech signal, is the signal length, is the delay amount.

[0032] For example, There is a voice signal ,calculate , ,

[0033]

[0034]

[0035]

[0036] Autocorrelation function Indicates the delay between the signal and itself The similarity after sampling points is calculated as follows: ① : Delay amount , which represents the correlation between the signal and itself when it is not delayed: and Multiplication, that is, each point is multiplied by itself and then summed

[0037] ② : Delay amount , which represents the correlation between the signal and itself after a delay of 1 sampling point: and Multiplication, that is, each point is multiplied by the next point and then summed

[0038] ③ : Delay amount , which represents the correlation between the signal and itself after being delayed by 2 sampling points: and Multiplication, that is, each point is multiplied by the next two points and then summed

[0039] Find out The first peak corresponds to , which is the preliminary estimated pitch period.

[0040] (2) Cepstrum analysis is used to extract the precise fundamental frequency, further improving the accuracy of pitch extraction. The calculation formula is:

[0041] in, is the fast Fourier transform, For its inverse transformation.

[0042] Continuing with the above signal as an example, conduct get , taking the logarithm we get , and then proceed get , The first significant peak position of is a more accurate estimate of the pitch period.

[0043] (3) Median filtering is used to smooth the pitch contour and obtain the smoothed pitch change curve. The calculation formula is as follows:

[0044] in, is the output sequence after filtering, is the input sequence, Indicates the median operation. is the filter window length.

[0045] Assume that the extracted base frequency sequence is Hz, using a median filter of length 3, and processing in sequence:

[0046]

[0047]

[0048] Finally, the smoothed pitch change curve y is obtained.

[0049] For the sound length information, in this embodiment, the hidden Markov model is first used to perform phoneme-level signature alignment on the speech, and then the dynamic time warping algorithm is used to calculate the precise duration of each phoneme. The specific steps are as follows: (1) Hidden Markov Model (HMM) Phoneme Alignment In this embodiment, HMM is used to perform mandatory alignment of speech at the phoneme level. Specifically, for example, there is a speech "Hello", and "Hello" is decomposed into a phoneme sequence: ; Establish an HMM model for each phoneme. In this embodiment, a 3-state left and right HMM is used; finally, the Viterbi algorithm is used to find the most likely state sequence, thereby determining the approximate boundary of each phoneme.

[0050] Among them, the recursive formula of the Viterbi algorithm is ,in It's time In state The maximum probability value of is Always in the state The maximum probability value of From the state Transfer to state The probability of express t The observed value at time, is in state Observed below The probability of In all possible previous states Select the largest probability value to find the most likely state transition path.

[0051] (2) Dynamic Time Warping (DTW) algorithm accurately calculates the duration of phonemes In this embodiment, for each phoneme, the DTW algorithm is used to accurately calculate its duration. Specifically, firstly, the feature sequence (such as MFCC feature) of the reference phoneme template and the phoneme to be tested is extracted; then the cumulative distance matrix is ​​constructed. ,in represents the minimum cumulative distance between the first i frames of the reference sequence and the first j frames of the sequence to be tested; then dynamically plan to fill the matrix : ,in It is Frame and The distance measurement of the frames; finally, the optimal path is traced back to determine the exact start and end times.

[0052] (3) Calculate the duration of the phoneme. The calculation formula is as follows: Duration (seconds) = (number of frames Frame shift) / sampling rate For example, if the sampling rate is 16kHz and the frame shift is 10ms, then The duration of a phoneme is (12*0.01) = 0.12 seconds.

[0053] This method can accurately calculate the duration of each phoneme, thereby obtaining detailed sound length information.

[0054] In addition to the syllable, pitch and length information, this embodiment also analyzes the formant of the signal.

[0055] For the resonance peak analysis, this embodiment mainly adopts a method combining linear predictive coding (LPC) and cepstrum analysis. The specific steps are as follows: (1) Preprocessing: Perform pre-emphasis and frame processing on the voice signal. The formula for pre-emphasis is: ,in It is usually taken as 0.97.

[0056] (2) LPC analysis: Perform LPC analysis on each frame of speech signal to obtain LPC coefficients. LPC analysis is based on the following model: , which attempts to use the previous The linear combination of sampling points is used to predict the value of the current sampling point, where For the current moment The speech signal value, The part represents the predicted value, for LPC coefficients, indicating the correlation between adjacent samples, is the prediction order (usually 8-12), For the front The speech signal value at a moment, is the prediction error, The goal of selecting the coefficients is to make the prediction error Minimum.

[0057] (3) Solve the LPC equation: The LPC equation is solved using the Levinson-Durbin recursive algorithm to obtain the LPC coefficients.

[0058] (4) Calculate the LPC spectrum: Calculate the LPC spectrum using the LPC coefficients: , where H(z) is the LPC transfer function (in frequency domain), is the characteristic of the linear prediction filter, is the prediction order (usually 8-12), are k LPC coefficients, is the delay operator of the z-transform, is the gain factor.

[0059] (5) Identify resonance peaks: Find local peaks in the LPC spectrum, these peaks are the formants. Usually focus on the first three formants F1, F2 and F3.

[0060] (6) Cepstrum analysis: The LPC spectrum is subjected to cepstrum analysis to improve the accuracy of formant estimation.

[0061] The calculation formula of the cepstrum C(n) is: .

[0062] (7) Smoothing: The obtained formant traces were median filtered to eliminate outliers and mutations.

[0063] In this step, the extracted speech features are closely related to oral and lip movements, which can provide an important basis for subsequent lip synchronization.

[0064] Step 3: Map the extracted speech features into lip shape change data.

[0065] In this embodiment, two methods of mapping speech features into lip shape change data are provided, including direct feature mapping and model mapping. In practical applications, either method can be selected to obtain lip shape change data.

[0066] (1) Direct feature mapping Direct feature mapping is the process of converting the extracted speech features directly into lip shape change data. This process is based on the knowledge of phonetics and physiology to establish a rule-based mapping system. The following is the specific mapping relationship between each speech feature and lip shape change data: ① Syllable mapping: Each syllable corresponds to a set of basic lip shape changes. For example, the vowel "a" usually corresponds to a larger opening, while "u" corresponds to a rounded lip shape.

[0067] ② Pitch mapping: Pitch changes mainly affect the vertical opening and closing degree of the lips. Higher pitches usually correspond to larger vertical opening degrees.

[0068] ③ Sound length mapping: Sound length determines the duration of lip shape change. Longer phonemes will have more lasting lip shape, while short phonemes will change lip shape faster.

[0069] ④ Formant mapping: Formant frequency is closely related to the shape of the oral cavity. The first formant (F1) mainly affects the opening and closing of the lips. The higher the F1, the wider the opening. The second formant (F2) mainly affects the roundness of the lips. The lower the F2, the rounder the lips.

[0070] ⑤ Transition sound mapping: Transition lip shape needs to be generated between adjacent phonemes, which is achieved through a linear interpolation algorithm.

[0071] ⑥ Co-articulation effect mapping: Taking into account the influence of adjacent phonemes, the lip shape of the current phoneme is adjusted. For example, when pronouncing "ku", the lip shape of "k" will be affected by the following "u", showing a slight lip rounding trend.

[0072] ⑦ Speech speed mapping: The overall speech speed affects the rate of lip shape change. According to the detected speech speed, the overall lip shape change is appropriately scaled or simplified.

[0073] The speech features extracted from the speech signal are converted in sequence according to the above mapping relationship to obtain the lip shape change data corresponding to the speech signal.

[0074] (2) Model mapping In this embodiment, an improved deep learning model is used to optimize the feature mapping process. The model combines the advantages of convolutional neural network (CNN) and long short-term memory network (LSTM), and is specifically used for mapping speech features to lip shape change data. The overall structure of the model is as follows: ① Input layer: receives the extracted speech features.

[0075] ②CNN module: Multi-layer convolutional layers are used to extract local features.

[0076] Pooling layer, used to reduce dimension and extract main features.

[0077] ③LSTM module: multi-layer bidirectional LSTM layer to capture long-term dependencies.

[0078] ④Fully connected layer: maps features to lip shape parameter space.

[0079] ⑤Output layer: Generates the final lip synchronization data.

[0080] The modules are connected through residual connections to facilitate information flow and gradient propagation.

[0081] On this basis, this model also introduces the following improvements: ① In the CNN module, multi-scale convolution kernels are used, including 1×1, 3×3, and 5×5 convolution kernels, to capture speech features of different time spans.

[0082] ②In the LSTM module, the self-attention mechanism is introduced to enable the model to dynamically focus on the important parts of the input sequence.

[0083] ③Add residual connections between CNN and LSTM modules, and between LSTM layers to effectively alleviate the gradient vanishing problem of deep networks.

[0084] By improving the model, it can better learn complex nonlinear mapping relationships and improve the accuracy of lip synchronization.

[0085] It should be noted that in addition to the main lip shape prediction task, the deep learning model proposed in this embodiment also introduces auxiliary tasks such as phoneme recognition to improve the generalization ability of the model by sharing the underlying feature representation.

[0086] Based on the constructed deep learning model, this embodiment adopts an adversarial training mechanism through the idea of ​​generative adversarial network (GAN) to further improve the authenticity and naturalness of the generated lip shape data.

[0087] Finally, through the trained deep learning model, the mapping of speech features to lip shape change data can be achieved.

[0088] Step 4: Apply the lip shape change data to the digital human model and perform real-time rendering to complete lip synchronization.

[0089] After the lip shape change data is obtained, it can be directly applied to the digital human model. In this embodiment, computer graphics are used to render the digital human model in real time to generate the final lip synchronization effect.

[0090] In one embodiment, in order to make the lip shape change more fluid, the lip shape change data obtained in step 3 is first smoothed before being applied to the digital human model to improve the continuity and naturalness of the action. In this process, an interpolation algorithm, such as linear interpolation, spline interpolation, etc., can be used to generate a smoother lip shape change.

[0091] The present invention accurately extracts speech features related to lip shape through speech feature extraction and feature mapping, and establishes an effective mapping relationship to achieve high-quality lip synchronization effect. The synchronization method proposed in the present invention has the following advantages: (1) High accuracy and naturalness: Direct analysis of speech signal features can more accurately capture subtle changes in pronunciation and generate more natural and smooth lip movements; (2) Strong language adaptability: It is suitable for a variety of language environments, and has particularly improved the processing effect of tonal languages ​​such as Chinese; (3) Good real-time performance: Optimized signal processing and mapping algorithms improve the speed of lip synchronization to meet the needs of real-time interaction; (4) Simple implementation: No complex phoneme recognition process is required, which reduces the difficulty of technical implementation and maintenance costs; (5) Enhanced user experience: Through more accurate and natural lip synchronization, the overall performance and natural interaction of digital humans are significantly improved.

[0092] In one embodiment, the present invention provides a voice-driven digital human lip synchronization system. By modularizing the working process of each step, each module can be optimized and upgraded separately, providing a flexible solution for different application scenarios.

[0093] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program includes a program code for executing the method shown in the flowchart.

[0094] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0095] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0096] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0097] As another aspect, the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the voice-driven digital human lip synchronization method described in the above embodiment.

[0098] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the voice-driven digital human lip synchronization method described in the above embodiment.

[0099] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0100] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software, or by combining software with necessary hardware. Therefore, the technical solution according to the implementation method of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation method of the present application.

[0101] It should be noted that in the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "setting" and "connection" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a direct connection, or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances; the drawings in the embodiments are used to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings herein can be arranged and designed in various different configurations.

[0102] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in the field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A digital human lip synchronization method based on speech drive, characterized in that: include: Acquire speech signals; Extract speech features related to lip shape changes in speech signals; Mapping the extracted speech features into lip shape change data; The lip shape change data is applied to the digital human model and rendered in real time to achieve lip synchronization.

2. The method for lip synchronization of digital human based on voice drive according to claim 1, characterized in that: The extracting of speech features related to lip shape changes in the speech signal specifically includes: Extract syllable information, pitch information, sound length information and formant from speech signals and integrate them into a comprehensive speech feature representation.

3. The method for lip synchronization of digital human based on voice drive according to claim 2, characterized in that: The syllable information extraction method comprises: Calculate the short-time energy and zero-crossing rate of speech information, and set the judgment threshold of syllable boundary; The beginning and end of each syllable are identified according to the judgment threshold to obtain the syllable information in the speech signal.

4. The method for lip synchronization of a digital human based on voice drive according to claim 2 or 3, characterized in that: The pitch information extraction method comprises: Perform autocorrelation analysis on speech signals to complete preliminary pitch estimation; Use cepstrum analysis to extract accurate fundamental frequency; The pitch contour is smoothed by median filtering to obtain a smoothed pitch change curve, i.e., pitch information.

5. The method for lip synchronization of digital human based on voice drive according to claim 2, characterized in that: The tone length information extraction method comprises: Forced alignment of speech at the phoneme level; Calculate the exact duration of each phoneme, i.e. the sound length information.

6. The method for lip synchronization of digital human based on voice drive according to claim 2, characterized in that: Linear predictive coding and cepstrum analysis are combined to complete the formant analysis.

7. The method for lip synchronization of digital human based on voice drive according to claim 2, characterized in that: The step of mapping the extracted speech features to lip shape change data specifically includes: Directly convert the extracted speech features into lip shape change data, or use a deep learning model to map the speech features into lip shape change data.

8. The method for lip synchronization of digital human based on voice drive according to claim 7, characterized in that: The directly converting the extracted speech features into lip shape change data specifically includes: Based on the knowledge of phonetics and physiology, speech features are mapped into lip shape change data, where each syllable corresponds to a set of basic lip shape changes; pitch maps the vertical opening and closing degree of the lip shape; sound length maps the duration of the lip shape change; resonance peaks map the opening and closing degree and the degree of lip rounding of the lip shape; transition lip shapes are generated between adjacent phonemes; co-articulation effects are used to map adjacent phonemes; and speech rate features characterize the rate of lip shape change.

9. The method for lip synchronization of digital human based on voice drive according to claim 7, characterized in that: The deep learning model is used to map speech features into lip shape change data, specifically including: A deep learning model based on convolutional neural network and long short-term memory network is established; wherein the deep learning model includes a connection input layer, a CNN module, an LSTM module, a fully connected layer and an output layer; the CNN module includes multiple convolutional layers and pooling layers, and the convolutional layer uses a multi-scale convolution kernel to capture speech features of different time spans; a self-attention mechanism is introduced in the LSTM module, and a residual connection is added between the CNN module and the LSTM module; Use adversarial training mechanism to complete the training of deep learning models; The trained model is used to map speech features to lip shape change data.

10. The method for lip synchronization of digital human based on voice drive according to claim 1, characterized in that: The method also includes interpolating the lip shape change data before applying the lip shape change data to the digital human model.

Images