A method for synthesizing interactive digital humans based on speech-driven artificial intelligence

Abstract

The present invention relates to the technical field of digital human synthesis, and specifically discloses an interactive digital human synthesis method based on voice-driven artificial intelligence, the method comprising: S1: collecting voice data submitted from different channels and platforms through a data acquisition unit, and preprocessing the input audio raw data, including voice filtering and voice feature digitization, extracting audio features through input voice signals based on an Audio2DHuman deep learning model, predicting facial key point motion weights and gesture and head animation parameters, and then generating facial expressions and smooth gestures and head movements that are precisely synchronized with the digital human language broadcast content, thereby achieving efficient driving and natural interaction of the digital human, thereby achieving synchronous mapping of the semantics and emotional expression contained in the voice to the rendering of the face, gestures, and body movements of a virtual human 3D model, thereby greatly enhancing the authenticity and professionalism of the digital human's emotions.

Description

Technical Field

[0001] The present invention relates to the technical field of digital human synthesis, and in particular to an interactive digital human synthesis method based on voice-driven artificial intelligence. Background Art

[0002] With the continuous development of science and technology, digital human generation technology has become a research hotspot in the fields of human-computer interaction, virtual reality and augmented reality. Technical research and theoretical innovation in the field of virtual digital humans have made continuous breakthroughs, and the application scope of virtual digital humans is gradually expanding, showing broad development prospects.

[0003] The development of virtual digital human applications can be roughly divided into three representative stages: 1.0 era. In this primary stage, the creation of digital humans mainly relies on two-dimensional hand-drawn or computer-generated animation CG technology to edit frame by frame and finally compile into a video. This process has a long production cycle and high cost; 2.0 era. With the rise of mixed reality technology, motion capture technology enables real-life movements to be mapped to 3D digital human models in real time. However, this stage has more stringent requirements for hardware equipment, resulting in a simultaneous increase in cost and implementation difficulty; 3.0 era. The integration of artificial intelligence technology has brought new changes to virtual digital human technology. The facial expressions of digital humans are aligned through audio content recognition and acoustic feature machine learning methods to achieve highly realistic driving effects.

[0004] Despite this, current digital human technology still mainly focuses on the alignment of speech and facial expressions, while research and application on how to synchronously map the semantics and emotional expressions contained in speech to the rendering of the face, gestures, and body movements of virtual human 3D models are still rare. Summary of the invention

[0005] The purpose of the present invention is to provide a method for synthesizing an interactive digital human based on voice-driven artificial intelligence to solve the following technical problems:

[0006] How to synchronously map the semantics and emotional expressions contained in speech to the facial, gesture, and body movement rendering of the virtual human 3D model.

[0007] The purpose of the present invention can be achieved through the following technical solutions:

[0008] A method for synthesizing an interactive digital human based on speech-driven artificial intelligence, the method comprising:

[0009] S1: The data collection unit collects voice data submitted from different channels and platforms, and pre-processes the input audio raw data, including voice filtering and voice feature digitization;

[0010] S2: Use the preprocessed data to train the Audio2DHuman model to generate the network structure and training data set required for synthetic content, including selecting a suitable deep learning model, adjusting model parameters, and performing model training and optimization;

[0011] S3: According to the received voice file, the parameters and configuration obtained by model training, 52 key frame weights of facial drive and 4 gestures and 4 head action animation drive data are automatically generated;

[0012] S4: Feedback the generated synthetic content to the user terminal for real-time rendering of the facial expressions, gestures, and head movements of the digital human 3D model in the user terminal.

[0013] Furthermore, the process of preprocessing the input audio raw data in S1 includes:

[0014] S11: Predict and filter human voices in audio data, and use the transformers audio classification model to perform binary classification tasks of valid and invalid human voices;

[0015] S12: Numericalization of speech features of audio files, compression and reconstruction of speech signals through LPC linear prediction coding, including audio framing, audio windowing, and signal reconstruction.

[0016] Furthermore, the network structure and training data set of the Audio2DHuman synthesis algorithm in S2 include:

[0017] Audio2DHuman's network structure module, training data set preparation module and Audio2DHuman model training module;

[0018] The network structure module of the Audio2DHuman includes a deep learning model and a model training loss calculation unit, and the deep learning model includes an audio feature learning layer, an emotion feature learning layer and an expression action mapping layer;

[0019] The model training loss calculation unit is used to optimize the loss function between the weight of the content generation output and the real sample based on the Audio2DHuman network model optimization goal, including weight term loss, motion frame loss, emotion regularization loss, gesture action classification loss, and head action classification loss.

[0020] Furthermore, the calculation process of the model training loss calculation unit includes:

[0021] By formula Calculate the mean square error between the weight value of each facial expression motion factor predicted by the network and the true value for the input x ;

[0022] By formula Calculate the mean square error between the weight change rate of the motion factor between two consecutive frames in all groups predicted by the network for the input x and the real frame;

[0023] By formula Calculate the mean of the sum of squares of all two consecutive frames of emotion feature vector change rates predicted by the network for input x ;

[0024] By formula Calculate the sum of the products of the logarithm of the probability of each category of gesture action predicted by the network for input x and the true value ;

[0025] By formula Calculate the sum of the products of the logarithm of the probability of each category of head movement predicted by the network for input x and the true value ;

[0026] Finally, the formula Calculate and obtain the minimum Loss value in the training phase;

[0027] in, The motion factor representing facial movements, is the serial number of the facial expression motion factor, is the true value of the weight of the i-th facial expression motion factor in the input x, is the weight value of the i-th facial expression motion factor predicted by the network for input x; Define the sign for the rate of change, which can be set based on empirical fitting; is the sequence number of the emotional state feature vector component, is a constant of 16, Represents the th component of the emotional state feature vector corresponding to the training sample; is the number of gesture action categories, For any gesture action category, is the one-hot encoding of the true category of the gesture action in the audio frame, is the predicted probability that the audio frame belongs to the gesture action category; is the number of head action categories, For any head movement, is the one-hot encoding of the true category of the head action in the audio frame, This audio frame belongs to the head action category The predicted probability of .

[0028] Further, the training data set production module includes an audio feature frame data processing unit and an audio frame label data production unit;

[0029] The audio feature frame data processing unit is used to extract audio features from the input audio through the LPC method, and form discrete values ​​to save as NPY files;

[0030] The audio frame label data production unit is used to produce labels for the audio frame by frame, which are divided into 52 key point weight data of the face, gesture action category data and head action category data.

[0031] Furthermore, the Audio2DHuman model training module includes a model index evaluation unit and an index effect feedback unit, wherein the model index evaluation unit is used to use root mean square error RMSE and Accuracy as evaluation indicators to evaluate the quality and accuracy of the generated weights;

[0032] The indicator effect feedback unit is used to use the stochastic gradient descent algorithm to perform model training, the learning rate is decayed according to the training rounds, and each round of training implements back propagation and parameter update based on the principle of minimizing the Loss value;

[0033] The evaluation process of the model indicator evaluation unit includes:

[0034] Root mean square error (RMSE) and accuracy are used as evaluation indicators to assess the quality and accuracy of the generated weights.

[0035] By formula Calculate the root mean square error ;

[0036] in, is the total number of frames, For any frame number, is the total number of weights, For any weight, For the The true value of the weight, For the The predicted value threshold obtained by the model with each weight;

[0037] And through the formula Calculate the accuracy of the classification effect of gestures and head movements ;

[0038] Among them, TP represents the number of samples correctly predicted as positive, FN represents the number of samples incorrectly predicted as negative, FP represents the number of samples incorrectly predicted as positive, and TN represents the number of samples correctly predicted as negative.

[0039] Furthermore, the process of automatically generating animation drive data in S3 includes:

[0040] S31: Traceability identification of data results, by assigning a unique identifier to each data instance to facilitate tracking and tracing during the process of data generation, processing, and dissemination;

[0041] S32: Generate a dictionary value for the text data item.

[0042] Furthermore, the process of feeding back the generated synthetic content to the user terminal in S4 includes:

[0043] The facial expression BlendShape weight data range received by the client from the server is between [0,1], while the 3D model BlendShape coefficient range in actual application is , so the weight values ​​need to be remapped;

[0044] By formula Calculate the remapped weight value.

[0045] in, is the maximum value of the BlendShape coefficient, is the minimum value of the BlendShape coefficient, The preset weight value.

[0046] Beneficial effects of the present invention:

[0047] (1) Based on the Audio2DHuman deep learning model, the present invention extracts audio features from the input speech signal, predicts the facial key point motion weights and gesture and head animation parameters, and then generates facial expressions and smooth gestures and head movements that are precisely synchronized with the digital human's language broadcast content, thereby achieving efficient driving and natural interaction of the digital human, thereby achieving the synchronous mapping of the semantics and emotional expression contained in the speech to the rendering of the virtual human's 3D model face, gestures, and body movements, thereby greatly enhancing the authenticity and professionalism of the digital human's emotions.

[0048] (2) The present invention predicts and filters the human voice in the audio data, digitizes the voice features of the audio file, and efficiently distinguishes between valid and invalid human voices, thereby ensuring the legality and compliance of the digital human input data, and synchronously maps the semantics and emotional expressions contained in the voice to the rendering of the virtual human 3D model's face, gestures, and body movements, thereby greatly enhancing the authenticity and professionalism of the digital human's emotions.

[0049] (3) Through the coordination of the audio feature learning layer, the emotion feature learning layer and the expression and action mapping layer, the present invention can generate expression animations containing new emotions through the features of facial speech expression weights and body movement weights learned from the training set, thereby greatly enhancing the authenticity and professionalism of the digital human's emotions. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The present invention will be further described below in conjunction with the accompanying drawings.

[0051] Figure 1 It is a flow chart of an interactive digital human synthesis method based on voice-driven artificial intelligence in the present invention;

[0052] Figure 2 It is a graph of the training loss of the Audio2DHuman model in the present invention;

[0053] Figure 3 Schematic diagram of the effect of left eye blink control coefficient 0-1 in the present invention;

[0054] Figure 4 This is a schematic diagram of the effect of the tongue extension control coefficient 0-1 in the present invention;

[0055] Figure 5 It is a schematic diagram of the distribution of digital human data weights on the face, gestures, and limbs in the present invention;

[0056] Figure 6 It is a flow chart of the algorithm of the Audio2DHuman model in the present invention. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] See also Figure 1 , Figure 6 As shown, in one embodiment, the present application provides a method for synthesizing an interactive digital human based on voice-driven artificial intelligence, the method comprising:

[0059] S1: The data collection unit collects voice data submitted from different channels and platforms, and pre-processes the input audio raw data, including voice filtering and voice feature digitization;

[0060] S2: Use the preprocessed data to train the Audio2DHuman model to generate the network structure and training data set required for synthetic content, including selecting a suitable deep learning model, adjusting model parameters, and performing model training and optimization;

[0061] S3: According to the received voice file, the parameters and configuration obtained by model training, 52 key frame weights of facial drive and 4 gestures and 4 head action animation drive data are automatically generated;

[0062] S4: Feeding back the generated synthetic content to the user terminal for real-time rendering of the facial expressions, gestures, and head movements of the 3D model of the digital human in the user terminal;

[0063] Through the above technical solution, when the method of this embodiment is used, the voice data submitted by different channels and platforms are first collected through the data collection unit. The data source may come from different channels and platforms. Then, a series of processing is performed on the input audio raw data. This step includes voice filtering and voice feature digitization to ensure the quality and validity of the data. The specific operations include human speech prediction, sensitive voice filtering, audio framing and windowing, LPC linear prediction coefficient and other operations to achieve compression and reconstruction preprocessing of the voice signal. Then, the preprocessed data is trained with the Audio2DHuman model. Through deep learning technology, The Audio2DHuman model uses the characteristics and rules of the deep learning data to generate the model parameters and configurations required for synthesizing content. This step includes selecting a suitable deep learning model, adjusting model parameters, and performing model training and optimization. The Audio2DHuman model includes an audio feature learning layer, an emotion feature learning layer, and an expression and action mapping layer. The audio feature learning layer extracts information using a fixed autocorrelation analysis function ACF and then uses a 5-layer convolution Conv+ReLU structure. The emotion feature learning layer consists of a 5-layer convolution Conv+ReLU structure. The expression and action mapping layer consists of three fully connected layers as weight outputs for the face, gesture, and head branches.

[0064] Afterwards, according to the received voice file, as well as the parameters and configuration obtained by model training, the acoustic network unit extracts the audio feature sequence that changes with time, and then uses the converter network unit to extract the feature encoding operation of the facial expression state from the audio feature sequence, and automatically generates 52 key frame weights driven by the face and 4 gestures, and 4 head movement animation driving data. The 52 key frame weights driven by the face include eyebrow movements, eye movements, nose movements, mouth movements, tongue movements, and cheek movements. These text data are combined into an easy-to-parse format. The file size usually depends on the audio duration and the frame size, which is generally 20KB / second. Finally, the generated synthetic content is returned to the user terminal for real-time rendering of the facial expressions, gestures, and head movements of the user terminal digital human 3D model, supporting PC web pages, mobile phone tablet APPs, VR all-in-one machines and other TCM digital human interactive scene applications;

[0065] Through such settings, based on the Audio2DHuman deep learning model, audio features are extracted from the input voice signal, and the facial key point movement weights and gesture and head animation parameters are predicted, thereby generating facial expressions and smooth gestures and head movements that are precisely synchronized with the digital human's language broadcast content, thereby achieving efficient driving and natural interaction of the digital human. In addition, by utilizing audio resources from a large amount of material to construct and annotate the digital human's facial expression and body movement data set, the authenticity and professionalism of the digital human's emotions are greatly enhanced.

[0066] The process of preprocessing the input audio raw data in S1 includes:

[0067] S11: Predict and filter human voices in audio data, and use the transformers audio classification model to perform binary classification tasks of valid and invalid human voices;

[0068] S12: Numericalization of speech features of audio files, compression and reconstruction of speech signals through LPC linear prediction coding, including audio framing, audio windowing, and signal reconstruction;

[0069] Through the above technical solution, when receiving raw audio data, the received voice data may include various types of Chinese voices, such as reading aloud, articles, conversation records, etc. These voice data may be in multiple languages ​​such as English and Chinese, and may contain noise interference, grammatical errors, pronunciation errors, etc. These voice data are input into the Audio2DHuman model after preprocessing and feature extraction. The judgment model supports voice classification models such as transformer and CNN. The transformers audio classification model is used to perform the binary classification task of effective human voice and invalid human voice. The audio classification model is built based on the wav2vec2-base pre-training model. Its core architecture is divided into four main modules: feature extraction, feature bridging, feature encoding and feature classification. In the feature extraction module, we use a 7-layer 1D convolutional network Conv1d to convert the original audio signal into an abstract high-level feature representation. The feature bridging module is responsible for the conversion of feature dimensions between the front and rear modules. The feature encoding module consists of 12 encoding layers, which can capture and learn the timing information and contextual relationship of the audio signal. The final classification module outputs the probability distribution of audio data belonging to different categories. During the model training process, we used public audio datasets including urban environmental sounds, Chinese speech datasets, and multi-domain Chinese speech datasets, and divided them into two categories: valid human voice and invalid human voice. After 60 cycles of iterative training, the accuracy of the model on the validation set reached 99.5%, which can effectively distinguish valid human voice from invalid human voice, ensuring the legality and compliance of the digital human input data;

[0070] The digitization of speech features is to extract meaningful features from speech signals to represent speech, so as to facilitate the analysis and application of speech signals. This algorithm adopts LPC linear predictive coding, and its principle is to assume that the current sample point in the speech signal can be expressed by the linear combination of the previous p sample points, so as to realize the compression and reconstruction of the speech signal, including three steps of audio framing, audio windowing, and signal reconstruction. First, the speech signal is framed (30ms per frame) and Hamming window processing is added. Then, the LPC autocorrelation coefficient of the frame signal is calculated using the autocorrelation function on each frame, and each frame of data is reconstructed by the LPC coefficient, and finally a segment of audio LPC coding is output;

[0071] Through such settings, valid human voices can be efficiently distinguished from invalid human voices, ensuring the legality and compliance of the data input by the digital human, thereby greatly enhancing the authenticity and professionalism of the digital human's emotions.

[0072] See also Figure 6 As shown, the network structure and training data set of the Audio2DHuman synthesis algorithm in S2 include:

[0073] Audio2DHuman's network structure module, training data set preparation module and Audio2DHuman model training module;

[0074] The network structure module of the Audio2DHuman includes a deep learning model and a model training loss calculation unit, and the deep learning model includes an audio feature learning layer, an emotion feature learning layer and an expression action mapping layer;

[0075] The model training loss calculation unit is used to optimize the loss function between the weight of the content generation output and the real sample based on the Audio2DHuman network model, including weight term loss, motion frame loss, emotion regularization loss, gesture action classification loss, and head action classification loss;

[0076] Through the above technical solution, the digital human driving model is used to generate the weight coefficients of the facial expressions and body animation categories of the 3D model. The network structure is shown in Table 1:

[0077]

[0078] Table 1: Network structure

[0079] The frame_num in the structure table is the number of audio frames. The model uses the audio LPC features as input and includes three modules:

[0080] Audio feature learning layer;

[0081] This layer consists of 5 layers of convolutional networks. After training, the network in this layer can extract short-term features in the audio: specific audio features such as phonemes, intonation, and stress;

[0082] Emotional feature learning layer;

[0083] The emotional feature learning layer is used to extract the associated features of adjacent sequence frames in time series and output the pronunciation feature map. Since pronunciation is not only related to the sound frequency, but also closely related to the speaker's emotions and type, the emotional feature learning layer also receives the emotional state extracted by training. The vector length is E (usually 16), which is directly connected to the pronunciation feature map and jointly learns the relationship with the expression with the pronunciation feature map. After being processed by the emotional feature learning layer, the final output is a feature vector with a length of (256+E), where 256 is the audio feature and E is the emotional feature. In the training stage, the input of the emotional vector is a random Gaussian distribution sampling of E dimensions. For the input samples with emotions, the feature vectors output by the emotional feature learning layer are stored during the training process. The E-dimensional feature vector is the emotional state contained in the training sample data that is automatically learned. The emotional feature vectors learned under different emotional data are combined to form an emotional state matrix. In the inference process, the user can select any set of vectors in the matrix as the emotional input to control the output expression to contain the specified emotion. Different emotional vectors can also be used as input by linearly combining them to generate expression animations containing new emotions.

[0084] Expression action mapping layer;

[0085] The (256+E)-dimensional feature vector output by the emotion feature learning layer is the feature of the facial expression weight and body movement weight learned from the training set. The obtained feature sequence is flattened and passed through a Dense layer feature mapping, and then passed through three fully connected layers as the output of the three branches. The three branches are:

[0086] Facial expression weight output (see frame_num*52 in the structure table), indicating that the facial expression follows the Apple facial ARKit standard;

[0087] Gesture action category weight output (see frame_num*4 in the structure table);

[0088] Head action category weight output (see frame_num*4 in the structure table);

[0089] Through such a setting, through the cooperation of the audio feature learning layer, the emotional feature learning layer and the expression and action mapping layer, the features of the facial speaking expression weights and body movement weights learned from the training set can be used to generate expression animations containing new emotions, thereby greatly enhancing the authenticity and professionalism of the digital human's emotions.

[0090] The calculation process of the model training loss calculation unit includes:

[0091] By formula Calculate the mean square error between the weight value of each facial expression motion factor predicted by the network and the true value for the input x ;

[0092] By formula Calculate the mean square error between the weight change rate of the motion factor between two consecutive frames in all groups predicted by the network for the input x and the real frame;

[0093] By formula Calculate the mean of the sum of squares of all two consecutive frames of emotion feature vector change rates predicted by the network for input x ;

[0094] By formula Calculate the sum of the products of the logarithm of the probability of each category of gesture action predicted by the network for input x and the true value ;

[0095] By formula Calculate the sum of the products of the logarithm of the probability of each category of head movement predicted by the network for input x and the true value ;

[0096] Finally, the formula Calculate and obtain the minimum Loss value in the training phase;

[0097] in, The motion factor representing facial movements, is the serial number of the facial expression motion factor, is the weighted true value of the i-th facial expression motion factor in the input x, is the weight value of the i-th facial expression motion factor predicted by the network for input x; Define the sign for the rate of change, which can be set based on empirical fitting; is the sequence number of the emotional state feature vector component, is a constant 16, Represents the th component of the emotional state feature vector corresponding to the training sample; is the number of gesture action categories, For any gesture action category, is the one-hot encoding of the true category of the gesture action in the audio frame, is the predicted probability that the audio frame belongs to the gesture action category; is the number of head action categories, For any head movement, is the one-hot encoding of the true category of the head action in the audio frame, The audio frame belongs to the head action category The predicted probability of

[0098] Through the above technical solution, by calculating the weight term, motion frame loss, emotion regularization loss, gesture action classification loss, head action classification loss, and through the formula The minimum Loss value in the training phase is calculated to minimize the cumulative and weighted sum of the five loss items, which means that the actions of the driving model tend to the expected values.

[0099] See also Figure 5 As shown, the training data set production module includes an audio feature frame data processing unit and an audio frame label data production unit;

[0100] The audio feature frame data processing unit is used to extract audio features from the input audio through the LPC method, and form discrete values ​​to save as NPY files;

[0101] The audio frame label data production unit is used to produce labels for the audio frame by frame, which are divided into 52 key point weight data of the face, gesture action category data and head action category data;

[0102] Through the above technical solution, when the audio feature frame data processing unit is working, the audio feature of the input audio is extracted by the LPC method, and the discrete values ​​are saved as NPY files. The specific steps are as follows:

[0103] In this project, the number of audio frames per second is set to 30. Therefore, for an audio with a length of S milliseconds, it contains TF= frames. When extracting LPC features, the audio features are defined to be mapped to Embedding_Dim∈(32,64)-dimensional data. Then, for the above audio with a length of S milliseconds, the discrete data dimension generated after reconstruction is (TF,32,64). According to the above method, feature mapping is performed on all audio files in the training set, and the discrete data of each audio is saved as an NPY format file;

[0104] When the audio frame label data production unit is working, it first needs to label the audio frame by frame, dividing it into facial expression weight data, gesture action category, and head action category. The specific steps are as follows:

[0105] Weight data of 52 key points on the face;

[0106] Use NVIDIA's Omniverse Audio2Face tool to convert audio data into BlendShape weight data based on ARkit standards. Enter the Audio2Face software and select the Asian model claire_bs_arkit with BlendShape weight under ARkit standards. Then select the audio file in the data set through the Audio2Face panel and automatically convert it into expression animation data. Then, through the BLENDSHAPE & JOINTCONVERSION module in the A2F Data Conversion panel, set the Playback FPS to 30, consistent with the 30 frames per second setting in the audio, select the input animation mesh and the output BlendShape mesh, and specify the export format as json to export the facial expression weight. This completes the conversion of audio into BlendShape weight data based on ARkit standards and 30 frames per second through the Omniverse audio2face platform. You can get dimensional data with a decimal value range of [0,1];

[0107] Gesture action category data;

[0108] Based on the basic speech segmentation method, a long audio is divided into several sub-audios according to the pauses in the tone of voice. Then, the audio emotion classification model is used to infer the emotion category corresponding to each sub-audio. The emotion categories are: natural, sad, praise, and passionate, corresponding to four types of gesture animations, 0: default action, 1: open palm, 2: thumbs up, 3: fist. By marking the corresponding gesture animation category on the audio data frame by frame, dimensional data can be obtained, and the value range is an integer between [0,3], as shown in Table 2:

[0109]

[0110] Table 2: Gesture action category data

[0111] Based on the basic speech segmentation method, a long audio is divided into several sub-audios according to the pauses in the tone of voice, and then the emotional category corresponding to each sub-audio is inferred through the audio emotion classification model. The emotional categories are: other, natural statement, contemplation, and passion, which correspond to 4 types of head animations, 0: default action, 1: slightly turn the head, 2: slightly lower the head, 3: slightly raise the head, and mark the corresponding head animation category on the audio data frame by frame to obtain the head action category dimension data, and the value range is an integer between [0,3]; as shown in Table 3;

[0112]

[0113] Table 3: Head movement category data

[0114] With this setting, we use code to parse and concatenate complete label data with the data dimension of ; the first 52 dimensions are decimals between [0,1], and the 53rd and 54th dimensions are integers between [0,3]. Similarly, all audio files in the training set are labeled according to the above steps and the label data of each audio is saved as an NPY format file, thereby realizing the production of audio frame label data.

[0115] See also Figure 2 As shown, the Audio2DHuman model training module includes a model index evaluation unit and an index effect feedback unit, and the model index evaluation unit is used to use root mean square error RMSE and Accuracy as evaluation indicators to evaluate the quality and accuracy of generated weights;

[0116] The indicator effect feedback unit is used to use the stochastic gradient descent algorithm to perform model training, the learning rate is decayed according to the training rounds, and each round of training implements back propagation and parameter update based on the principle of minimizing the Loss value;

[0117] The evaluation process of the model indicator evaluation unit includes:

[0118] Root mean square error (RMSE) and accuracy are used as evaluation indicators to assess the quality and accuracy of the generated weights.

[0119] By formula Calculate the root mean square error ;

[0120] in, is the total number of frames, For any frame number, is the total number of weights, For any weight, For the The true value of the weight, For the The predicted value threshold obtained by the model with each weight;

[0121] And through the formula Calculate the accuracy of the classification effect of gestures and head movements ;

[0122] Among them, TP represents the number of samples correctly predicted as positive, FN represents the number of samples incorrectly predicted as negative, FP represents the number of samples incorrectly predicted as positive, and TN represents the number of samples correctly predicted as negative;

[0123] Through the above technical solution, the root mean square error The root mean square error threshold For comparison, if Greater than , then the training effect is judged to be poor and further training is needed. Otherwise, if the training effect is judged to be good, further training is not needed. The accuracy of the classification effect of gesture and head movements is calculated. Accuracy with preset For comparison, if Greater than , then the training accuracy is high and no further training is needed; otherwise, the training effect is poor and further training is needed;

[0124] The model is trained using the gradient descent algorithm. The learning rate decays according to the number of training rounds. Each round of training implements back propagation and parameter updates based on the principle of minimizing the Loss value. From the loss curve, it can be seen that the loss begins to show a stable trend when the model iteration reaches Steps=50,000, and continues throughout the entire training cycle until the end of the model iteration, indicating that the model has converged and can accurately convert audio features into corresponding facial expression weights, head movements, and gestures. On the test data set, the RMSE is lower than 0.15 and the Accuracy is higher than 0.95. Figure 2 shown.

[0125] See also Figure 3 , Figure 4 As shown, the process of automatically generating animation driving data in S3 includes:

[0126] S31: Traceability identification of data results, by assigning a unique identifier to each data instance to facilitate tracking and tracing during the process of data generation, processing, and dissemination;

[0127] S32: Generate a dictionary value of the text data item;

[0128] Through the above technical solution, when this embodiment is used, a unique identifier is first assigned to each data instance to track and trace the source during the data generation, processing, and dissemination process. Our methods for providing traceability identification for algorithm-synthesized content include adding timestamp identification, digital fingerprints, steganography, and explicit identification methods;

[0129] The number is composed of the encrypted hash of the data content. By comparing the digital fingerprint, the authenticity and integrity of the data can be verified in the subsequent tracking and tracing process. The steganography of audio processing technology can hide the identification information in the frame data dimension. By parsing and returning the frame weight identification information, tracking and tracing are achieved. The displayed identification position is the JSON field identification, including the request timestamp, return timestamp, audio frame length, data data signature, and algorithm version label. By giving each data instance a unique identifier, it can be tracked and traced during the data generation, processing, and dissemination process.

[0130] Then generate the text data item dictionary value, the data item dictionary value framework is shown in Table 4:

[0131]

[0132]

[0133] Table 4: Facial expression weight values

[0134] Facial expressions include: mouth, tongue, nose, eyebrows, eyes, cheeks, a total of 52 groups of weight values, ranging from 0 to 1. Gesture animations include: default action (hands naturally hanging down and closing in front of the abdomen), open palms (palms facing up, thumbs naturally open), thumbs up (thumbs up), fists (clenched fists, one or two hands placed on the chest), representing natural, sad, praise, and passionate emotional expressions respectively. Head animations include: default action (looking straight ahead), slightly turning the head (randomly facing left or right), slightly lowering the head, and slightly raising the head. The weight coefficients have an effect on blinking and sticking out the tongue. Figure 3 , Figure 4 shown.

[0135] The process of feeding back the generated synthetic content to the user terminal in S4 includes:

[0136] The facial expression BlendShape weight data received by the client from the server ranges from [0,1], while the 3D model BlendShape coefficient range in actual applications is , so the weight values ​​need to be remapped;

[0137] By formula Calculate the remapped weight value.

[0138] in, is the maximum value of the BlendShape coefficient, is the minimum value of the BlendShape coefficient, is the preset weight value;

[0139] Through the above technical solution, through the formula After calculating the remapped weight values, the BlendShape coefficient range of the 3D model in actual applications can be updated to ensure the accuracy of the gesture animation and head animation driving model, thereby greatly enhancing the authenticity and professionalism of the digital human's emotions.

[0140] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A method for synthesizing interactive digital humans based on voice-driven artificial intelligence, characterized in that: The method comprises: S1: The data collection unit collects voice data submitted from different channels and platforms, and pre-processes the input audio raw data, including voice filtering and voice feature digitization; S2: Use the preprocessed data to train the Audio2DHuman model to generate the network structure and training data set required for synthetic content, including selecting a suitable deep learning model, adjusting model parameters, and performing model training and optimization; S3: According to the received voice file, the parameters and configuration obtained by model training, 52 key frame weights of facial drive and 4 gestures and 4 head action animation drive data are automatically generated; S4: Feeding back the generated synthetic content to the user terminal for real-time rendering of the facial expressions, gestures, and head movements of the 3D model of the digital human in the user terminal; The process of preprocessing the input audio raw data in S1 includes: S11: Predict and filter human voices in audio data, and use the transformers audio classification model to perform binary classification tasks of valid and invalid human voices; S12: Numericalization of speech features of audio files, compression and reconstruction of speech signals through LPC linear prediction coding, including audio framing, audio windowing, and signal reconstruction; The network structure and training data set in S2 include: Audio2DHuman's network structure module, training data set preparation module and Audio2DHuman model training module; The network structure module of the Audio2DHuman includes a deep learning model and a model training loss calculation unit, and the deep learning model includes an audio feature learning layer, an emotion feature learning layer and an expression action mapping layer; The model training loss calculation unit is used to optimize the loss function between the weight of the content generation output and the real sample based on the Audio2DHuman network model, including weight term loss, motion frame loss, emotion regularization loss, gesture action classification loss, and head action classification loss; The calculation process of the model training loss calculation unit includes: By formula Calculate the mean square error between the weight value of each facial expression motion factor predicted by the network and the true value for the input x ; By formula Calculate the mean square error between the weight change rate of the motion factor between two consecutive frames in all groups predicted by the network for the input x and the real frame; By formula Calculate the mean of the sum of squares of all two consecutive frames of emotion feature vector change rates predicted by the network for input x ; By formula Calculate the sum of the products of the logarithm of the probability of each category of gesture action predicted by the network for input x and the true value ; By formula Calculate the sum of the products of the logarithm of the probability of each category of head movement predicted by the network for input x and the true value ; Finally, the formula Calculate and obtain the minimum Loss value in the training phase; in, The motion factor representing facial movements, is the serial number of the facial expression motion factor, is the weighted true value of the i-th facial expression motion factor in the input x, is the weight value of the i-th facial expression motion factor predicted by the network for input x; Define the sign for the rate of change, set based on empirical fit; is the sequence number of the emotional state feature vector component, is a constant 16, The first Quantity; is the number of gesture action categories, For any gesture action category, is the one-hot encoding of the true category of the gesture action in the audio frame, is the predicted probability that the audio frame belongs to the gesture action category; is the number of head action categories, For any head movement, is the one-hot encoding of the true category of the head action in the audio frame, The audio frame belongs to the head action category The predicted probability of .

2. The interactive digital human synthesis method based on voice-driven artificial intelligence according to claim 1, characterized in that: The training data set production module includes an audio feature frame data processing unit and an audio frame label data production unit; The audio feature frame data processing unit is used to extract audio features from the input audio through the LPC method, and form discrete values ​​to save as NPY files; The audio frame label data production unit is used to produce labels for the audio frame by frame, which are divided into 52 key point weight data of the face, gesture action category data and head action category data.

3. The interactive digital human synthesis method based on voice-driven artificial intelligence according to claim 2, characterized in that: The Audio2DHuman model training module includes a model index evaluation unit and an index effect feedback unit, wherein the model index evaluation unit is used to use root mean square error RMSE and Accuracy as evaluation indicators to evaluate the quality and accuracy of the generated weights; The indicator effect feedback unit is used to use the stochastic gradient descent algorithm to perform model training, the learning rate is decayed according to the training rounds, and each round of training implements back propagation and parameter update based on the principle of minimizing the Loss value; The evaluation process of the model indicator evaluation unit includes: Root mean square error (RMSE) and accuracy are used as evaluation indicators to assess the quality and accuracy of the generated weights. By formula Calculate the root mean square error ; in, is the total number of frames, For any frame number, is the total number of weights, For any weight, For the The true value of the weight, For the The predicted value threshold obtained by the model with each weight; And through the formula Calculate the accuracy of the classification effect of gestures and head movements ; Among them, TP represents the number of samples correctly predicted as positive, FN represents the number of samples incorrectly predicted as negative, FP represents the number of samples incorrectly predicted as positive, and TN represents the number of samples correctly predicted as negative.

4. The interactive digital human synthesis method based on voice-driven artificial intelligence according to claim 3 is characterized in that: The process of automatically generating 52 key frame weights of facial drive and 4 gestures and 4 head action animation drive data in S3 includes: S31: Traceability of data results, by assigning a unique identifier to each data instance so that it can be tracked and traced during the data generation, processing and dissemination process; S32: Generate a dictionary value for the text data item.

5. The interactive digital human synthesis method based on voice-driven artificial intelligence according to claim 4 is characterized in that: The process of feeding back the generated synthetic content to the user terminal in S4 includes: The facial expression BlendShape weight data received by the client from the server ranges from [0,1], while the 3D model BlendShape coefficient range in actual applications is , so the weight values ​​need to be remapped; By formula Calculate the remapped weight value. in, is the maximum value of the BlendShape coefficient, is the minimum value of the BlendShape coefficient, The preset weight value.

Images