A digital human generation method based on sound driving

Abstract

The application relates to the technical field of digital human generation, in particular to a digital human generation method based on sound driving, which comprises the following steps: multi-modal data acquisition and preprocessing; audio-expression feature extraction and synchronization degree calculation; expression dynamic stability modeling and analysis; model optimization and generation based on reinforcement learning. The application aims to generate a digital human image that is highly consistent with audio and has natural and smooth emotional expression.

Description

Technical Field

[0001] This application relates to the field of digital human generation technology, specifically to a voice-driven digital human generation method. Background Technology

[0002] Thanks to advancements in artificial intelligence and computer graphics, voice-driven digital human generation is a crucial research area in human-computer interaction and virtual reality. It converts input sound signals into corresponding lip movements, facial expressions, and body gestures in real time, ensuring consistency between sound and visual representation. Previously, digital human animation relied on hand-drawn animation or offline text acquisition, which was time-consuming, labor-intensive, and lacked realism. However, recent breakthroughs in deep learning for speech recognition, semantic understanding, and image generation have enabled voice-driven technology to evolve from phoneme-based lip-syncing to emotion and rhythm-based responses, thus more powerfully showcasing the expressiveness and realism of digital humans.

[0003] Current voice-driven digital human generation systems cannot extract and translate the "paralinguistic information" representing tone and emotion from the raw audio signal, including variations in pitch, speech rate, rhythm, and intensity. They also cannot accurately map this complex signal information into subtle, coordinated movements of facial muscles. As a result, the generated digital human's expressions do not adapt well to the speech, exhibiting problems such as emotional misalignment (e.g., using laughter to express sadness), inappropriate intensity (expressions are either too static or disorganized), and discontinuous dynamics. This results in a lack of realism and emotional depth in the digital human, giving users a very artificial feeling. To date, most current digital human generation models only utilize a static "phoneme-lip shape" mapping method, remaining severely lacking in suprasegmental features, making it difficult to achieve the slow, gradual, natural, and appropriate changes in facial expressions like a human. Summary of the Invention

[0004] To address the aforementioned technical problems, this application provides a voice-driven digital human generation method to solve the existing issues.

[0005] This application proposes a voice-driven digital human generation method using the following technical solution:

[0006] One embodiment of this application provides a voice-driven digital human generation method, which includes the following steps:

[0007] The synchronously recorded audio signal and the facial video stream are processed in a synchronized frame-by-frame manner.

[0008] The MFCC feature vector in each frame of audio signal and the facial action unit (AU) feature in the facial video stream are extracted, and the similarity and distance between the two are analyzed after dimensionality reduction to calculate the audio expression synchronization degree of each frame.

[0009] Clustering methods were used to divide all audio facial expression synchronization into high, medium and low synchronization ranges, and the dynamic stability of facial expressions was calculated by the cluster label of each synchronization range and the difference between the actual value and the predicted value of the element in the cluster.

[0010] The dynamic stability of facial expressions is incorporated into the reward function of the reinforcement learning training process to optimize the output digital human facial expression parameters based on the input audio signal-facial video stream, thereby assisting in the generation of digital humans from facial expressions and audio.

[0011] Preferably, the MFCC feature vector is extracted by using Mel frequency cepstral coefficients (MFCC) for each frame of audio signal, including 13-dimensional MFCC coefficients and their first and second order differences with adjacent frames, and spliced ​​to form a 39-dimensional MFCC feature vector.

[0012] Preferably, the facial motion unit features are extracted using the OpenFace tool and include intensity values ​​of 44 facial motion units.

[0013] Preferably, the method for calculating the audio expression synchronization is as follows:

[0014] Calculate the mean cosine similarity between the MFCC principal component eigenvectors and the AU principal component eigenvectors of all frames;

[0015] Calculate the DTW distance between the MFCC principal component feature vector and the AU principal component feature vector for each frame;

[0016] The ratio of the mean to the DTW distance is used as the audio-expression synchronization degree for each frame.

[0017] Preferably, the dimensionality reduction is performed by principal component analysis to reduce the dimensionality to a 3-dimensional principal component vector.

[0018] Preferably, the dynamic stability of the facial expression is calculated in the following way:

[0019]

[0020] Where B represents the dynamic stability of facial expressions, and N represents the number of clusters. Let i be the clustering label of the i-th cluster. It is the mean of the absolute differences between the actual values ​​and the predicted values ​​of all elements in the i-th cluster.

[0021] Preferably, the method for determining each synchronization segment is as follows: using a clustering method to divide all audio expression synchronization into three clusters; and according to the mean of the elements within the cluster from large to small, the clusters are sequentially divided into high, medium, and low synchronization segments.

[0022] Preferably, the method for determining the cluster labels is as follows: the clusters corresponding to the high, medium and low synchronization segments are labeled as 3, 2 and 1 respectively.

[0023] Preferably, the reward function after incorporating facial expression dynamic stability into the reward function of the reinforcement learning training process is defined as:

[0024]

[0025] in This is the basic reward item, with a value of L1 is the L1 loss between the intensity value of the generated facial action unit (AU) and the intensity value of the real face AU; L2 is the L2 loss between the generated lip shape vertex and the real lip shape vertex; B is the expression dynamic stability; and λ is the preset weight coefficient.

[0026] Preferably, the predicted value is obtained by predicting all audio facial expression synchronization as input to the AR model.

[0027] This application has at least the following beneficial effects:

[0028] 1. This application addresses the problem that it is difficult to effectively extract and map paralinguistic information (such as pitch, speech rate, and rhythm) in audio signals, leading to a disconnect between digital human facial expressions and speech content, and emotional misalignment. It uses the Mel frequency cepstral coefficient (MFCC) extraction algorithm and principal component analysis (PCA) method to construct audio-expression synchronization degree A, which reflects the temporal matching quality and fluency between audio features and facial action units (AU). This solves the problems of insufficient micro-matching between facial expressions and audio and macro-temporal distortion, significantly improving the synchronization and naturalness of digital human facial expressions and speech.

[0029] 2. This application addresses the problems of inconsistent facial expressions, emotional misalignment, and stiffness, which make it difficult to maintain a smooth transition between expressions. It uses an autoregressive model (AR) and a K-means clustering algorithm to calculate the dynamic stability B of facial expressions, which reflects the degree of fluctuation and phased patterns of facial expression synchronization over time. This eliminates the influence of abnormal fluctuations and prediction biases, thereby accurately identifying the intervals of facial expression disjointness and providing a quantitative basis for optimizing facial expression generation.

[0030] 3. This application introduces the dynamic stability of facial expressions, B, as a reward function into a reinforcement learning framework, and combines it with generative adversarial networks (GANs) for adversarial training and multi-timescale optimization. This enables dynamic adjustment and iterative improvement of the digital human generation model, effectively minimizing the dynamic instability of facial expressions and solving problems such as inappropriate facial expression intensity and dynamic jumps. Ultimately, it generates digital human images that are highly consistent with the audio and have natural and fluent emotional expression, improving the realism and user experience in scenarios such as high-end customer service and virtual idols. Attached Figure Description

[0031] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 A flowchart of a voice-driven digital human generation method provided in this application. Detailed Implementation

[0033] To further illustrate the technical means and effects adopted by this application to achieve the intended inventive purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a voice-driven digital human generation method proposed in this application. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0035] The following description, in conjunction with the accompanying drawings, details a specific scheme for a voice-driven digital human generation method provided in this application.

[0036] One embodiment of this application provides a voice-driven digital human generation method.

[0037] Specifically, the following voice-driven digital human generation method is provided; please refer to [link / reference]. Figure 1 The method includes the following steps:

[0038] Step 1: Collect and process multimodal data.

[0039] The actors' audio was recorded synchronously in a professional anechoic chamber using high-fidelity condenser microphones with configuration; three front-facing high-speed infrared cameras were used to record the actors' facial information; according to the script requirements, the actors delivered speech that included expressions, different emotions, speech speeds, and tones, and optical markers were used to mark the position of the actors' brows as reference points for post-production calibration.

[0040] The collected raw data undergoes fine preprocessing: the audio signal is first pre-emphasized and then denoised, and then framed in 25ms increments (frame shift 10ms) to obtain the framed audio signal sequence; the video stream is analyzed by facial action unit analysis using the OpenFace 2.0 toolkit, and the intensity values ​​of 44 facial action units (AUs) defined in the Facial Action Coding System (FACS) are extracted to obtain the time series of AU intensities that are strictly time-aligned with the audio frames.

[0041] Step 2: Extract audio-expression features and calculate their synchronization.

[0042] Voice-driven digital human generation struggles to accurately capture and match the paralinguistic information (pitch, rhythm, and intensity variations) contained in audio signals to corresponding facial expressions. Therefore, when mapping the spoken components of a given speech from various parameters using digital human generation algorithms, some core elements cannot be restored, which can easily lead to mismatches between the generated voice and lip movements, resulting in emotional distortion, intensity imbalance, or dynamic incoherence.

[0043] In voice-driven digital human generation, MFCC features are used to obtain voice factors including lip movements and emotions. Therefore, this application utilizes the Mel Frequency Cepstral Coefficients (MFCC) algorithm. The framed audio signal sequence is input into the algorithm, with a frame length of 25ms and a frame shift of 10ms. The 13-dimensional MFCC coefficients, along with their first and second-order differences, are extracted from each frame. By concatenating the 13-dimensional MFCC coefficients and their first and second-order differences with those of adjacent frames, a 39-dimensional MFCC feature vector is formed.

[0044] The first-order difference captures the dynamic characteristics of speech, reflecting short-term dynamic changes in audio features; the second-order difference reflects higher-level dynamic patterns of audio features, further enhancing the descriptive ability of dynamic features compared to the first-order difference. The 39-dimensional feature vector formed by concatenating these three features can simultaneously capture both static and dynamic information of the audio, more comprehensively describing the phoneme content and suprasegmental features (such as speech rate, intonation, and emotion), and more accurately mapping the relationship between audio and audio analogous units (AUs). The MFCC feature vector output by the algorithm is the spectral envelope of the audio at the Mel scale, simulating the nonlinear frequency perception of the human ear. Therefore, it can fully describe the phoneme content and suprasegmental features (such as speech rate and intonation), which is beneficial for describing non-speech features in lip movement capture of action elements.

[0045] However, high-dimensional MFCC features and high-dimensional facial action units have high mapping complexity and large computational cost.

[0046] To further improve the efficiency of feature association, Principal Component Analysis (PCA) was used to reduce the dimensionality of facial action units, with time series of 44 core AU intensities as input values. The input was linearly transformed by PCA to obtain its principal component vectors, and the total variance was retained at 95%. The output was a 3-dimensional principal component feature vector of the facial action unit. These three dimensions correspond to the three principal components of the dimensionality-reduced facial action unit, representing the main change patterns of facial expressions. In the process of digital human expression generation, redundancy and collinearity of AUs were removed, and complex muscle movements were simplified into a small number of expression primitives. This makes it easier for the expression features generated by the dimensionality-reduced digital human to be combined with the emotional expression of the digital human's pronunciation, thereby improving mapping efficiency and generalization effect.

[0047] Then, using the same method, with the 39-dimensional MFCC feature vector as input, the PCA algorithm is used to output the 3-dimensional MFCC principal component feature vector.

[0048] For each frame, the cosine similarity S between the 3D MFCC principal component feature vector and the 3D AU principal component feature vector at the current acquisition time is calculated. The cosine similarity measures the degree of directional agreement between the audio feature pattern and the facial expression feature pattern in the t-th frame. The Dynamic Time Warping (DTW) algorithm is introduced to calculate the DTW distance D between the 3D MFCC principal component feature vector and the 3D AU principal component score vector.

[0049] From the above discussion, the expression for the audio-expression synchronization degree A of each frame is as follows:

[0050]

[0051] In the definition of A, The mean cosine similarity between the MFCC principal component feature vectors and the AU principal component feature vectors of all frames is used to characterize the micro-matching degree between the audio stream and the facial expression stream at each time step, i.e., whether the spectral features of the audio stream are similar to the cooperative patterns of the facial decomposition units of the facial expression stream. D represents the DTW distance between the MFCC principal component feature vectors and the AU principal component feature vectors of each frame, indicating the ease of temporal alignment between the two vectors. This represents the following effect and smoothness of the rhythm of facial expression changes on the prosodic rhythm of speech. A larger distance D indicates greater temporal differences in the facial animation compared to speech, such as delay, lead, and distortion. Therefore, the average instantaneous matching accuracy is obtained while overcoming the difficulty of temporal distortion. Thus, the larger the A value, the higher the degree to which the digital human in the system can completely match the audio stream within a frame, and the higher the temporal consistency between facial expressions and speech streams can be demonstrated.

[0052] Step 3: Modeling and analysis of facial expression dynamic stability.

[0053] Because voice-driven digital humans present a challenge in that the pitch, speed, and rhythm of the vocal cords in an audio signal are non-linearly correlated with the subtle movements in facial expressions, it is difficult to systematically achieve stable and smooth facial expression transitions. When synthesizing digital humans, this results in problems such as inconsistent facial expressions, emotional misalignment, and stiff facial expressions, leading to poor realism and a poor user experience.

[0054] The solution to this problem is to use an autoregressive (AR) model. Using the time series data of index A as input, the AR model is set to order 3. Three frames effectively reflect the dynamic relationship between two consecutive frames without causing the model to overfit. Finally, the AR model's predicted values ​​are output.

[0055] The predicted value represents the average of historical synchronization values, reflecting the temporal evolution of facial expression synchronization. In voice-driven digital human generation, AR models are used to model temporal dependencies to capture abnormal changes in inter-frame synchronization, thereby identifying the moments when facial expressions and audio do not match. For example, when the residual is large, it indicates that there is a mismatch between the digital human and a certain part of the sound in a certain frame, i.e., there is an abnormal moment, and the facial expression at this time will obviously become illogical.

[0056] Introducing AR model residuals can detect fluctuations but cannot identify phased patterns in facial expression synchronization. To address this issue, time-series data of index A is used as input data. K-means clustering is used to divide index A into three categories (k=3), corresponding to high, medium, and low synchronization segments (this setting covers the most typical facial expression patterns). Specifically, the clusters are mapped to high, medium, and low synchronization segments according to the mean of elements within each cluster from largest to smallest, and their cluster labels are labeled as 3, 2, and 1 respectively. The cluster labels and cluster centers are output. The output cluster labels are used to reflect the overall stability pattern of digital human facial expressions under audio-driven conditions. During voice-controlled digital human generation, K-means clustering can distinguish between relatively stable and variable synchronization segments by grouping them. For example, the high synchronization segment can indicate that the digital human's facial expression is relatively consistent with the audio emotion, which can be used to locate the interval where emotional misalignment occurs, while the low synchronization segment is the facial expression part that needs to be corrected.

[0057] Based on the above analysis, the formula for calculating the dynamic stability B of facial expressions is:

[0058]

[0059] Where N is the number of clusters, Let i be the clustering label of the i-th cluster. Let be the mean of the absolute differences between the actual and predicted values ​​of all elements in the i-th cluster; and let the clusters be labeled as 3, 2, and 1 in descending order of the mean of the elements within the cluster.

[0060] When calculating the dynamic stability B of facial expressions, use This indicates the level of facial expression synchronization in each audio segment within the cluster. KMeans clustering segments the audio facial expression synchronization A time series, and the label value of each cluster (high, medium, and low synchronization correspond to 3, 2, and 1 respectively) reflects the matching quality level between the digital human's facial expressions and audio drivers at different time periods. Combined with... The mean residual between the actual values ​​of all elements within the cluster and the predicted values ​​of the AR model (i.e., the difference between the observed and expected values ​​of synchronization A) can quantify the temporal fluctuation of facial expression synchronization and the magnitude of prediction error over a given period. A weighted average is then used to calculate the dynamic stability B of the facial expression. B is used to calculate the dynamic stability of the expression, employing a weighted average method that uses synchronization level and residual fluctuation as weights to obtain the coherence and reliability of the digital human's expression under audio-driven conditions. A higher B indicates that the digital human's expression is less likely to achieve a smooth transition when the emotion and tone of the audio signal change. This leads to a larger synchronization deviation between the expression and the audio. Discontinuities exist in both high and low synchronization segments, resulting in the digital human's expression easily exhibiting a fixed state and a flat, emotionless surface, failing to accurately express the character's emotions.

[0061] Step 4: Utilize reinforcement learning to optimize and generate models.

[0062] Current digital human generation models fall short in extracting and mapping "paralinguistic information" from audio signals, resulting in digital humans exhibiting stiff facial expressions, misplaced emotions, and inconsistent dynamics. Specifically, traditional methods mostly employ a static "phoneme-lip shape" mapping approach, failing to effectively map the emotional and intonation information carried by suprasegmental features such as pitch, speech rate, and rhythm. Furthermore, they lack a large number of high-quality, high-precision "audio-expression" paired samples. Moreover, the algorithmic models have weak generalization ability when faced with complex emotional semantics and nonlinear mapping relationships between facial action units, thus failing to accurately simulate natural and fluent human facial expression transitions. This leads to discrepancies between the generated digital facial expressions and the spoken content, affecting the realism of real-time interaction and the emotional expressiveness of expressions, making them unsuitable for applications requiring a high degree of emotional interaction, such as high-end customer service or virtual idols.

[0063] Based on the above considerations, a hybrid optimization strategy is adopted to improve the final expressiveness of the digital human's facial expressions. The process of the hybrid optimization strategy is as follows:

[0064] First, a reinforcement learning model learns how to generate temporally coherent and naturally transitioning facial expression sequences. The reward function optimization uses the basic L1 / L2 reconstruction loss and the facial expression dynamic stability B as inputs, and the updated policy network parameters are output through proximal policy optimization (PPO), driving the model to actively improve facial expression coherence while reducing prediction errors. Next, adversarial training is introduced to continuously improve the detail and realism of the realistic generator. Adversarial training uses real facial expression sequences and generator-generated facial expression sequences as inputs to the discriminator, and optimizes the generator through feedback on the authenticity of its output, forcing the generated expressions to approximate real data in detail and dynamics. Finally, multi-timescale training is combined to balance the short-term accuracy and long-term consistency of digital human expressions. Multi-timescale training simultaneously inputs short-term and long-term audio-facial expression data, calculates the loss at different time granularities and backpropagates it, outputting a robust generation model that can ensure accurate lip-sync and maintain natural emotional continuity. This model is used to output the optimized facial expression parameters, thereby systematically synthesizing highly realistic and expressive digital human expressions.

[0065] To ensure the accuracy of generated expressions while addressing the issues of stiff and inconsistent dynamic transitions, facial dynamic stability (B) is added to the reward function of the reinforcement learning training process. Specifically, the reinforcement learning training process takes audio signals and facial video streams as inputs, and outputs the digital human's facial expression parameters through a policy network. The reinforcement learning training process is well-known to those skilled in the art and will not be elaborated upon here. The reward function of the improved reinforcement learning training process is defined as follows:

[0066]

[0067] in This is the basic reward item, with a value of L1 is the L1 loss between the generated facial action unit (AU) parameters and the real face AU parameters, L2 is the L2 loss between the generated lip vertices (or lip features) and the real lip shape, B is the expression dynamic stability, and λ is a preset weight coefficient with a value range of 0.4 to 0.7. In this embodiment, the value is specifically 0.5, which is used to balance the penalty intensity of expression dynamic stability B. The calculation of L1 loss and L2 loss is a well-known technique in the reward function of the expression parameter prediction module, and the specific process will not be described in detail.

[0068] Finally, after training across multiple time scales to output the final facial expression parameters, these parameters are used as driving signals and input into the 3D facial model of the digital human. This model is built using blendshape technology, where each AU parameter corresponds to the displacement weight of a specific facial muscle group. Rendering engines (such as Unity, Unreal Engine, or specialized graphics APIs) deform the model mesh, adjust textures, and calculate lighting and shadows in real time based on these weight values, thus transforming abstract numerical parameters into concrete, realistic facial expression images with skin texture and three-dimensionality. This completes the final transformation from data to visual representation, allowing users to intuitively see a digital human image that is highly synchronized with audio and expresses emotions naturally.

[0069] The above technical features constitute the preferred embodiment of this application, which has strong adaptability and the best implementation effect. Unnecessary technical features can be added or removed according to actual needs to meet the needs of different situations.

Claims

1. A voice-driven digital human generation method, characterized in that, The method includes the following steps: The synchronously recorded audio signal and the facial video stream are processed in a synchronized frame-by-frame manner. The MFCC feature vector in each frame of audio signal and the facial action unit (AU) feature in the facial video stream are extracted, and the similarity and distance between the two are analyzed after dimensionality reduction to calculate the audio expression synchronization degree of each frame. Clustering methods were used to divide all audio facial expression synchronization into high, medium and low synchronization ranges, and the dynamic stability of facial expressions was calculated by the cluster label of each synchronization range and the difference between the actual value and the predicted value of the element in the cluster. The dynamic stability of facial expressions is incorporated into the reward function of the reinforcement learning training process to optimize the output digital human facial expression parameters based on the input audio signal-facial video stream, thereby assisting in the generation of digital humans from facial expressions and audio. The method for calculating the audio expression synchronization is as follows: calculate the mean of the cosine similarity between the MFCC principal component feature vector and the AU principal component feature vector of all frames. Calculate the DTW distance between the MFCC principal component feature vector and the AU principal component feature vector for each frame; The ratio of the mean to the DTW distance is used as the audio-expression synchronization degree of each frame; The predicted value is obtained by using the synchronization of all audio expressions as input to the AR model.

2. The voice-driven digital human generation method as described in claim 1, characterized in that, The MFCC feature vector is extracted by using Mel frequency cepstral coefficients (MFCC) for each frame of audio signal. It includes 13-dimensional MFCC coefficients and their first and second order differences with adjacent frames, which are then concatenated to form a 39-dimensional MFCC feature vector.

3. The voice-driven digital human generation method as described in claim 1, characterized in that, The facial motion unit features were extracted using the OpenFace tool, including intensity values ​​for 44 facial motion units.

4. The voice-driven digital human generation method as described in claim 1, characterized in that, The dimensionality reduction is achieved by principal component analysis to reduce the dimensionality to 3-dimensional principal component vectors.

5. The voice-driven digital human generation method as described in claim 1, characterized in that, The dynamic stability of the facial expression is calculated in the following way: Where B represents the dynamic stability of facial expressions, and N represents the number of clusters. Let i be the clustering label of the i-th cluster. It is the mean of the absolute differences between the actual values ​​and the predicted values ​​of all elements in the i-th cluster.

6. The voice-driven digital human generation method as described in claim 5, characterized in that, The method for determining each synchronization segment is as follows: use a clustering method to divide all audio expression synchronization into three clusters; according to the mean of the elements within the cluster from large to small, divide the clusters into high, medium and low synchronization segments in sequence.

7. The voice-driven digital human generation method as described in claim 6, characterized in that, The method for determining the cluster labels is as follows: the clusters corresponding to the high, medium and low synchronization degree segments are labeled as 3, 2 and 1 respectively.

8. The voice-driven digital human generation method as described in claim 1, characterized in that, The reward function, after incorporating facial expression dynamic stability into the reward function of the reinforcement learning training process, is defined as follows: in This is the basic reward item, with a value of L1 is the L1 loss between the intensity value of the generated facial action unit (AU) and the intensity value of the real face AU; L2 is the L2 loss between the generated lip shape vertex and the real lip shape vertex; B is the expression dynamic stability; and λ is the preset weight coefficient.

Images