A method and system for driving virtual human using voice signal

Abstract

The present invention belongs to the field of speech signal processing technology, and discloses a method and system for driving a virtual human with a speech signal, the method comprising: resampling the original audio signal data to obtain a speech signal, and solving the linear prediction coefficient based on a dynamic frame division strategy and a linear prediction analysis technology to obtain a speech feature sequence of the speech signal; using a dynamic feature enhancement technology to enhance the speech feature sequence, and obtaining an enhanced feature sequence based on the processing result, and using the enhanced feature sequence as input to construct an animation sequence prediction model; obtaining animation sequence parameters based on the output result of the animation sequence prediction model, and combining the animation sequence parameters with a virtual human role model to perform facial action synthesis and rendering operations. The present invention greatly simplifies the pre-processing process and improves the ease of use and applicability in the driving process.

Description

Technical Field

[0001] The present invention relates to the technical field of voice signal processing, and in particular to a method and system for driving a virtual human with a voice signal. Background Art

[0002] With the rise of technologies such as virtual reality (VR), augmented reality (AR) and the metaverse, digital humans have received widespread attention as a medium for human-computer interaction. By driving virtual humans with voice, the user's voice content and tone can be instantly converted into the virtual human's mouth shape, facial expressions, and head posture changes, which can greatly enhance the naturalness and immersion of the interaction. However, existing technologies often have problems such as lack of dynamic context and single output when driving virtual humans with voice. This not only leads to redundant and complex systems, but also fails to achieve information sharing and collaborative optimization between tasks at the feature level.

[0003] The existing technologies for driving virtual humans specifically include methods such as sequence-to-sequence (Seq2Seq) models and generative adversarial networks (GANs), but they have the following limitations when applied: Sequence-to-sequence (Seq2Seq) models: This method uses an encoder-decoder structure to encode the input speech features into hidden representations and then decodes them into corresponding lip parameter sequences. For example, some studies use bidirectional LSTM as an encoder and combine it with an attention mechanism to improve the accuracy of lip generation. Long short-term memory networks (LSTMs) or gated recurrent units (GRUs) are used to map speech sequences to facial expression sequences, but model training is difficult and prone to overfitting; Generative adversarial network (GAN) methods: GANs consist of a generator and a discriminator. The generator attempts to generate realistic lip sequences, while the discriminator evaluates whether the generated results are real. Some advanced methods combine GANs with conditional inputs (such as speech features) to achieve high-fidelity 3D lip generation. GANs are used to generate realistic facial expressions, but GAN training is unstable and the model complexity is high.

[0004] Based on the existing technology, it usually relies on large neural network models with huge parameters, which are difficult to run on resource-constrained devices. It also mostly uses fixed windows to extract static speech features (such as MFCC, Mel spectrum), ignoring the dynamic changes and contextual associations of speech signals on the time axis, resulting in the mismatch between the lip shape and expression generation and the actual speech rhythm and pronunciation dynamics. At the same time, most methods only learn for a single task and fail to utilize the correlation between multiple tasks, limiting the generalization ability of the model. This not only leads to redundant and complex systems, but also fails to achieve information sharing and collaborative optimization between tasks at the feature level, resulting in the lack of details in the facial expressions of the generated digital humans, which cannot truly reflect the emotional and semantic information of the speech.

[0005] Therefore, how to provide a method and system for driving virtual humans with voice signals is a problem that needs to be solved urgently. Summary of the invention

[0006] The embodiment of the present invention provides a method and system for driving a virtual human using a voice signal, so as to solve the problem in the prior art that the emotion and semantic information of the voice cannot be truly reflected.

[0007] In order to have a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not a general review, nor is it intended to identify key / important components or to describe the scope of protection of these embodiments. Its only purpose is to present some concepts in a simple form as a prelude to the detailed description that follows.

[0008] According to a first aspect of an embodiment of the present invention, a method for driving a virtual human using a voice signal is provided.

[0009] In one embodiment, the method of driving a virtual person with a voice signal includes:

[0010] The original audio signal data is resampled to obtain a speech signal, and the linear prediction coefficient is solved based on the dynamic frame division strategy and linear prediction analysis technology to obtain the speech feature sequence of the speech signal;

[0011] The speech feature sequence is enhanced by using dynamic feature enhancement technology, an enhanced feature sequence is obtained based on the processing result, and the enhanced feature sequence is used as input to construct an animation sequence prediction model;

[0012] The animation sequence parameters are obtained based on the output results of the animation sequence prediction model, and the animation sequence parameters are combined with the virtual human character model to perform facial action synthesis and rendering operations.

[0013] In one embodiment, the original audio signal data is resampled to obtain a speech signal, and the linear prediction coefficient is solved based on the dynamic frame strategy and the linear prediction analysis technology to obtain the speech feature sequence of the speech signal, including:

[0014] The original audio signal is subjected to anti-aliasing filtering and difference processing using an open-source audio data tool to obtain a resampled speech signal. The frame length of the speech signal is determined by calculating the characteristic short-time energy based on a dynamic framing strategy.

[0015] The speech signal is divided according to the frame length and the frame shift, and after the division is completed, the speech signal is subjected to windowing processing using the Hamming window technique to obtain a preprocessed speech signal;

[0016] The linear relationship between speech signals and speech samples is estimated based on linear prediction analysis technology, and the minimum mean square error criterion is used to construct and solve a linear equation group to obtain linear prediction coefficients as the speech feature sequence of the speech signal.

[0017] In one embodiment, the linear relationship between the speech signal and the speech sample is estimated based on the linear prediction analysis technology, and the linear equations are constructed and solved using the minimum mean square error criterion to obtain the linear prediction coefficients, as the speech feature sequence of the speech signal includes:

[0018] The speech signal is represented as a linear combination of samples, the prediction error term of the linear combination of samples is analyzed, and the linear relationship between the speech signal and the speech sample is estimated based on the prediction error term and the linear prediction analysis technology;

[0019] Based on the linear relationship result, the autocorrelation function of the speech signal is obtained, and the linear equation group is constructed and solved by using the autocorrelation function and the minimum mean square error criterion, and the linear prediction coefficient is obtained according to the solution result;

[0020] The obtained sets of linear prediction coefficients are combined as speech feature parameters of the speech signal, and a speech feature sequence is obtained according to the speech feature parameters.

[0021] In one embodiment, the expression of the autocorrelation function is:

[0022] ;

[0023] In the formula, represents the autocorrelation function, N Indicates the frame length. k Indicates the index of the frame, represents the speech signal, Indicates forward offset k The speech signal after the frame, P Indicates the order of the linear prediction model.

[0024] In one embodiment, the speech feature sequence is enhanced by using dynamic feature enhancement technology, an enhanced feature sequence is obtained based on the processing result, and the enhanced feature sequence is used as input to construct an animation sequence prediction model, including:

[0025] Based on the dynamic feature enhancement technology, the time series length and feature dimension of the speech feature sequence changing with time are analyzed, and the time series length, feature dimension and speech feature sequence are spliced ​​to obtain an enhanced feature sequence;

[0026] Using a facial capture device to record speech collaborative multimodal data including audio and facial animation, and exporting facial data after fine-tuning the speech collaborative multimodal data;

[0027] The enhanced feature sequence is used as the input layer to build a prediction architecture model, and the facial data is used as the training set to train the prediction architecture model network to obtain the animation sequence prediction model.

[0028] In one embodiment, the enhanced feature sequence is used as the input layer to construct a prediction architecture model, and the facial data is used as a training set to perform network training on the prediction architecture model, and the animation sequence prediction model is obtained, including:

[0029] The enhanced feature sequence is set as the input layer for feature map generation, and an encoder is constructed based on several groups of convolutional layers. The encoder is used to extract local features to obtain a temporal feature sequence.

[0030] A fully connected layer is used to map the temporal feature sequence to the animation sequence parameter space to generate a decoder, and the output layer structure is determined based on the decoder. The input layer, encoder, decoder and output layer are combined to obtain a prediction architecture model.

[0031] After preprocessing the facial data, the data is divided into training set, validation set and test set according to the proportion. The training set is used to verify the prediction output of the prediction architecture model, and the gradient of the loss to the prediction architecture model parameters is calculated based on the prediction output results.

[0032] Based on the gradient results, the optimizer is used to update the prediction architecture model parameters, and the optimization algorithm is used to update the hyperparameters. The performance of the updated prediction architecture model is evaluated based on the validation set. The number of training iterations is determined based on the evaluation results to obtain the animation sequence prediction model.

[0033] In one embodiment, the expression of the prediction architecture model is:

[0034] ;

[0035] In the formula, represents the parameters of the animation sequence predicted by the prediction architecture model, f represents the overall mapping function of the prediction architecture model, z represents the enhanced speech features of the input, i represents the parameter set of the prediction architecture model, represents the encoder function, represents the decoder function, i ε and i D They represent the encoder convolutional layer parameters and the decoder fully connected layer parameters respectively.

[0036] In one embodiment, the expression of the optimization algorithm is:

[0037] ;

[0038] In the formula, represents the mean absolute error, N batch represents the number of samples in each batch, Indicates l The actual animation sequence parameters of the group training set, Indicates l The parameters of the animation sequence predicted by the group training set.

[0039] In one embodiment, obtaining animation sequence parameters based on the output result of the animation sequence prediction model, and combining the animation sequence parameters with the virtual human character model to perform facial action synthesis and rendering operations include:

[0040] Obtain animation sequence parameters based on the output result of the animation sequence prediction model, apply the animation sequence parameters to the virtual human 3D character model, and render the virtual human 3D character model using rendering technology;

[0041] Facial expression driving and head posture are realized based on the rendering results, and facial expressions are smoothed and voice operations are performed in combination with facial skeleton animation technology to realize voice signal driving of virtual people.

[0042] According to a second aspect of an embodiment of the present invention, a system for driving a virtual human using a voice signal is provided.

[0043] In one embodiment, the system for driving a virtual person with a voice signal includes:

[0044] The speech feature sequence acquisition module is used to resample the original audio signal data to obtain the speech signal, and solve the linear prediction coefficient based on the dynamic frame division strategy and linear prediction analysis technology to obtain the speech feature sequence of the speech signal;

[0045] A sequence prediction model building module is used to enhance the speech feature sequence using dynamic feature enhancement technology, obtain an enhanced feature sequence based on the processing result, and use the enhanced feature sequence as input to build an animation sequence prediction model;

[0046] The virtual facial rendering and synthesis module is used to obtain animation sequence parameters based on the output results of the animation sequence prediction model, and combine the animation sequence parameters with the virtual human character model to perform facial action synthesis and rendering operations.

[0047] According to a third aspect of an embodiment of the present invention, a computer device is provided.

[0048] In one embodiment, the computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0049] According to a fourth aspect of embodiments of the present invention, a computer-readable storage medium is provided.

[0050] In one embodiment, the computer-readable storage medium stores a computer program, and the computer program implements the steps of the above method when executed by a processor.

[0051] The technical solution provided by the embodiment of the present invention may have the following beneficial effects:

[0052] 1. The present invention simplifies the input processing and does not require complex signal preprocessing. The voice signal only needs to be converted into a fixed mono signal before entering the network. There is no need to perform complex signal processing steps such as dynamic range compression, noise reduction or pre-emphasis filter, which greatly simplifies the pre-processing process and improves the ease of use and applicability during the driving process.

[0053] 2. The present invention adopts a dynamic framing strategy to improve the accuracy of speech feature extraction and optimize computational efficiency. By integrating dynamic convolution and multi-head self-attention mechanism, the accuracy of capturing dynamic changes in speech is significantly improved, thereby making the virtual human's lip shape and expression generation closer to real-time speech changes.

[0054] 3. The present invention adopts a multi-task learning framework to learn expression and action generation tasks, and utilizes the correlation between tasks to integrate facial parameter prediction and head posture prediction into the same model. Multi-task learning realizes feature sharing, improves generation consistency and reduces computational redundancy, so that high performance can be maintained even with a small amount of data, making the development process more efficient. There is no need to rely on a large amount of training data input, which greatly reduces the cost of data collection and processing. Facial animation parameters and head posture parameters can be directly generated from speech without the need for intermediate representation or manual feature engineering, thus simplifying the training process.

[0055] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0057] Figure 1 is a flow chart of a method for driving a virtual person with a voice signal according to an exemplary embodiment;

[0058] Figure 2 is a principle block diagram of a system for driving a virtual person with a voice signal according to an exemplary embodiment;

[0059] Figure 3is a schematic diagram of an implementation according to an exemplary embodiment;

[0060] Figure 4 The figure is a schematic diagram showing the structure of a computer device according to an exemplary embodiment. DETAILED DESCRIPTION

[0061] The following description and accompanying drawings fully illustrate the specific embodiments of this article so that those skilled in the art can practice them. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. The scope of the embodiments of this article includes the entire scope of the claims, as well as all available equivalents of the claims. Herein, the terms "first", "second", etc. are only used to distinguish one element from another, without requiring or implying any actual relationship or order between these elements. In fact, the first element can also be called the second element, and vice versa. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the structure, device or equipment including a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also include elements inherent to such structure, device or equipment. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the structure, device or equipment including the elements. Each embodiment is described in a progressive manner herein, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the embodiments can be referred to each other.

[0062] The terms "longitudinal", "lateral", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc. in this document indicate the orientation or position relationship based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing this document and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In the description of this document, unless otherwise specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a mechanical connection or an electrical connection, it can also be the internal communication of two elements, it can be a direct connection, or it can be an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0063] As used herein, the term "plurality" means two or more than two, unless otherwise specified.

[0064] In this document, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B means: A or B.

[0065] In this article, the term "and / or" is a description of the association relationship between objects, indicating that three relationships may exist. For example, A and / or B means: A or B, or, A and B.

[0066] It should be understood that, although the various steps in the flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the figure may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0067] Each module in the device or system of the present application can be implemented in whole or in part by software, hardware, or a combination thereof. The above modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to the above modules.

[0068] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0069] Figure 1 An embodiment of the method of driving a virtual human using a voice signal of the present invention is shown.

[0070] In this optional embodiment, the method of driving a virtual person by a voice signal includes:

[0071] Step S101, resampling the original audio signal data to obtain a speech signal, and solving the linear prediction coefficient based on the dynamic frame division strategy and the linear prediction analysis technology to obtain the speech feature sequence of the speech signal;

[0072] Step S103, using dynamic feature enhancement technology to enhance the speech feature sequence, obtaining an enhanced feature sequence based on the processing result, and using the enhanced feature sequence as input to construct an animation sequence prediction model;

[0073] Step S105, obtaining animation sequence parameters based on the output result of the animation sequence prediction model, and combining the animation sequence parameters with the virtual human character model to perform facial action synthesis and rendering operations.

[0074] In this optional embodiment, when the original audio signal data is resampled to obtain a speech signal, and the linear prediction coefficient is solved based on the dynamic framing strategy and linear prediction analysis technology to obtain the speech feature sequence of the speech signal, the audio data open source tool can be used to perform anti-aliasing filtering and difference processing on the original audio signal to obtain the resampled speech signal, and the frame length of the speech signal is determined by calculating the characteristic short-time energy based on the dynamic framing strategy; the language signal is divided according to the frame length and frame shift, and after the division is completed, the speech signal is windowed using the Hamming window technology to obtain the preprocessed speech signal; the linear relationship between the speech signal and the speech sample is estimated based on the linear prediction analysis technology, and the minimum mean square error criterion is used to construct a linear equation group for solving to obtain the linear prediction coefficient as the speech feature sequence of the speech signal.

[0075] In this optional embodiment, when estimating the linear relationship between the speech signal and the speech sample based on the linear prediction analysis technology, and using the minimum mean square error criterion to construct and solve a linear equation group to obtain linear prediction coefficients as the speech feature sequence of the speech signal, the speech signal can be represented as a linear combination of samples, the prediction error term of the sample linear combination is analyzed, and the linear relationship between the speech signal and the speech sample is estimated based on the prediction error term and the linear prediction analysis technology; the autocorrelation function of the speech signal is obtained based on the linear relationship result, and the linear equation group is constructed and solved using the autocorrelation function and the minimum mean square error criterion, and the linear prediction coefficients are obtained according to the solution result; several groups of linear prediction coefficients obtained by the solution are combined as the speech feature parameters of the speech signal, and the speech feature sequence is obtained according to the speech feature parameters.

[0076] In this optional embodiment, when the speech feature sequence is enhanced by using dynamic feature enhancement technology, an enhanced feature sequence is obtained based on the processing result, and the enhanced feature sequence is used as input to build an animation sequence prediction model, the time series length and feature dimension of the speech feature sequence that change with time can be analyzed based on the dynamic feature enhancement technology, and the time series length, feature dimension and the speech feature sequence are spliced ​​to obtain an enhanced feature sequence; speech collaborative multimodal data including audio and facial animation is recorded by using a facial capture device, and facial data is exported after fine-tuning the speech collaborative multimodal data; a prediction architecture model is constructed using the enhanced feature sequence as the input layer, and the prediction architecture model is trained on the network using the facial data as a training set to obtain an animation sequence prediction model.

[0077] In this optional embodiment, when the enhanced feature sequence is used as the input layer to construct a prediction architecture model, and the facial data is used as the training set to perform network training on the prediction architecture model to obtain the animation sequence prediction model, the enhanced feature sequence can be set as the feature map to generate the input layer, and an encoder can be constructed based on several groups of convolutional layers, and the encoder can be used to extract local features to obtain a temporal feature sequence; the temporal feature sequence can be mapped to the animation sequence parameter space using a fully connected layer to generate a decoder, and the output layer structure can be determined based on the decoder, and the input layer, encoder, decoder and output layer can be merged to obtain the prediction architecture model; the facial data is preprocessed and divided into a training set, a validation set and a test set in proportion, the training set is used to verify the prediction output of the prediction architecture model, and the gradient of the loss to the prediction architecture model parameters is calculated based on the prediction output result; the prediction architecture model parameters are updated using an optimizer based on the gradient result, and the hyperparameters are updated using an optimization algorithm, the performance of the updated prediction architecture model is evaluated based on the validation set, and the number of training iterations is determined based on the evaluation result to obtain the animation sequence prediction model.

[0078] In this optional embodiment, when obtaining animation sequence parameters based on the output results of the animation sequence prediction model, and combining the animation sequence parameters with the virtual human character model to perform facial action synthesis and rendering operations, the animation sequence parameters can be obtained based on the output results of the animation sequence prediction model, and the animation sequence parameters can be applied to the virtual human three-dimensional character model, and the virtual human three-dimensional character model can be rendered using rendering technology; facial expression driving and head posture are realized based on the rendering results, and facial expressions are smoothed and voice operated in combination with facial skeleton animation technology to realize voice signal driving of the virtual human.

[0079] Figure 2 An embodiment of a system for driving a virtual person using a voice signal according to the present invention is shown.

[0080] In this optional embodiment, the system for driving a virtual person with a voice signal includes:

[0081] The speech feature sequence acquisition module 201 is used to resample the original audio signal data to obtain a speech signal, and solve the linear prediction coefficient based on the dynamic frame division strategy and the linear prediction analysis technology to obtain the speech feature sequence of the speech signal;

[0082] The sequence prediction model building module 203 is used to enhance the speech feature sequence using the dynamic feature enhancement technology, obtain the enhanced feature sequence based on the processing result, and use the enhanced feature sequence as input to build the animation sequence prediction model;

[0083] The virtual face rendering and synthesis module 205 is used to obtain animation sequence parameters based on the output results of the animation sequence prediction model, and combine the animation sequence parameters with the virtual human role model to perform facial action synthesis and rendering operations.

[0084] like Figure 3 As shown, the method of driving a virtual human with voice proposed in this embodiment realizes high-precision digital human driving on resource-constrained devices based on the existing technology that often has problems such as lack of dynamic context and single output when driving a virtual human with voice, and has important application value and broad market prospects. It mainly includes the following steps:

[0085] Step 1: Speech preprocessing;

[0086] The input speech signal needs to be preprocessed first to unify the sampling rate of the audio data to ensure the consistency of the model processing. In this embodiment, the open source tool librosa library (a dedicated Python library for audio and music analysis) is used to resample the audio to 16kHz. The sampling process generally involves two steps: 1. Anti-aliasing filtering; 2. Interpolation or downsampling (the resampling process is relatively simple, just call the librosa.resmple function directly).

[0087] Step 2: Speech feature extraction module;

[0088] 1. Dynamic framing: The traditional fixed-length framing method is difficult to adapt to the diversity of speech signals. It is changed to a dynamic framing strategy, which can adjust the frame length according to the short-time energy (STE) of the speech signal characteristics. The high-energy segment (speech activity area) uses a shorter frame length to enhance the time resolution; the low-energy segment (silent area) uses a longer frame length to reduce the calculation cost. This strategy can improve the accuracy of speech feature extraction and optimize the calculation efficiency. The specific calculation method is as follows:

[0089] ;

[0090] In the formula, E [ n ] represents the energy of the audio signal frame, G Indicates the audio frame length, x [ n ] represents the speech signal, n Indicates the current frame. x Indicates the amplitude value of the speech signal. m Indicates the index of a sampling point in the current frame.

[0091] In addition, the high energy segment (speech activity area) can divide the continuous speech signal into short time frames, because the speech signal can be considered to be stable in a short time, and each frame usually contains 10 to 30 milliseconds of data. s ( n ) According to the fixed frame length N and frame shift MIn order to reduce the changes between frames and ensure that there are overlapping cross-regions between adjacent frames during sampling, a smooth speech signal can be obtained. k Frame signal for:

[0092] ;

[0093] In the formula, s ( n ) represents the original speech signal, k Indicates the frame index (frame number), N Indicates the frame length. In this embodiment, the frame length is 520 (the number of samples contained in each frame). M Indicates frame shift. In this embodiment, the frame shift is 260 (the moving step between frames, usually M < N , there is overlap).

[0094] 2. Windowing: Reduce the boundary effect between frames and improve the accuracy of spectrum estimation. Multiply each frame signal by a window function. w ( n ), in this embodiment, the window function used is the Hamming Window (Hamming Window, HW), and the signal after windowing for:

[0095] ;

[0096] Definition of Hamming window:

[0097] ;

[0098] 3. Linear prediction analysis: If the short-term frame signal in the dynamic framing is stable, the mouth shape will not change significantly during one frame, that is, the length of one frame will not be longer than the length of one phoneme. The difference in speech phonemes is directly reflected in the resonance frequency of the speech signal. The determining factor of the resonance frequency is the value of the autocorrelation coefficient. Based on this principle, this embodiment uses an optimization strategy, that is, omitting most of the processing procedures and directly using the autocorrelation coefficient as the representation of the instantaneous resonance peak information. This method is highly consistent with the convolutional network, because the convolutional layer can effectively learn the estimation method of the instantaneous spectrum in a specific frequency band. Through linear prediction, the current speech signal sample is estimated to be different from the previous one. Q The linear relationship between the samples is used to extract the linear prediction coefficients. Assuming that the signal of the current frame can be expressed as Q A linear combination of samples, plus a prediction error term , represents the contribution of the sound source signal (such as vocal cord vibration) to the current sample, reflecting the part of the speech signal that is not captured by the vocal tract model. This parameter is mainly used for speech synthesis:

[0099] ;

[0100] In the formula, It means predicting the current frame signal. a k represents the linear prediction coefficient (need to be solved), P Indicates the order of the prediction model (usually P is 12 to 16, the same below), represents the prediction error (residual or activation signal).

[0101] The method to solve the linear prediction coefficient is to use the minimum mean square error criterion to construct and solve a set of linear equations. The commonly used method is the autocorrelation method. The process is as follows:

[0102] (1) Calculate the autocorrelation function :

[0103] ;

[0104] In the formula, represents the autocorrelation function, N Indicates the frame length. k Indicates the index of the frame, represents the speech signal, Indicates forward offset k The speech signal after the frame, P Indicates the order of the linear prediction model.

[0105] (2) Construct a system of linear equations:

[0106] ;

[0107] Or it can be expressed in matrix form:

[0108] ;

[0109] In the formula, R express P × P dimensional autocorrelation matrix (AM), whose elements are ,in i and j Respectively represent the row and column indices of the equations in the linear equation system, and the value range is 1- P , a represents the linear prediction coefficient vector, P ×1 dimension, r represents the autocorrelation vector (AV), P ×1 dimension, a iFor the i Autocorrelation coefficients are obtained, and the linear prediction coefficients are obtained by solving the linear equations.

[0110] (4) Extracting LPC (Linear Prediction Coefficient): k The autocorrelation linear prediction coefficients are used as speech feature parameters for subsequent processing. Assume that the obtained time series LPC feature sequence is X ={ x 1, x 2, ..., x T},and ,in, d x represents the feature dimension, T Indicates the length of the time series.

[0111] Step 3: Dynamic feature enhancement module;

[0112] Assume that the LPC feature sequence obtained in step 2 is X ={ x 1, x 2, ..., x T}, after the dynamic feature enhancement module, the enhanced feature sequence is obtained: Z ={ z 1, z 2, ..., z T},in:

[0113] ;

[0114] in, L Represents the half window length of the convolution kernel in the time axis direction, Conv (·) represents the convolution feature extraction of local segments, Attention (·) represents the multi-head self-attention mechanism, which is used to dynamically weight and merge features globally to obtain enhanced features that are more sensitive to temporal context.

[0115] Step 4: Multi-task learning (network training);

[0116] 1. Training data preparation: Professional actors use professional facial capture equipment to record clear lip-synced speech collaborative multimodal data (including audio + facial animation). Animators refine the exported animations and then export high-precision facial data (metahuman skeletal controller controlRig parameter animation sequence frames, the same below). The audio is saved in wav format, mono, with a sampling rate of no less than 16kHz.

[0117] 2. Model architecture:

[0118] (1) Input layer: The enhanced feature sequence obtained by the dynamic feature enhancement module in step 3 is used as a 2D expression (feature map) input for subsequent convolutional layers.

[0119] (2) Encoder: It consists of several convolutional layers, which mainly extract local context features and learn to extract short-term features useful for facial animation: intonation and specific phonemes, etc., as well as learn head posture features to obtain the latent space representation of the temporal feature sequence;

[0120] (3) Decoder: It consists of fully connected layers, and the temporal features extracted by the encoder step are mapped to the controlRig parameter space;

[0121] (4) Output layer: Outputs the controlRig parameters of the metahuman skeletal controller (60 for the face and 2 for the head posture in this embodiment), which are expressed as follows:

[0122] ;

[0123] In the formula, represents the parameters of the animation sequence predicted by the prediction architecture model, f represents the overall mapping function of the prediction architecture model, z represents the enhanced speech features of the input, i represents the parameter set of the prediction architecture model, represents the encoder function, which consists of convolutional layers with parameters i ε , represents the decoder function, which consists of a fully connected layer with parameters i D , i ε and i D Represent the encoder convolutional layer parameters and the decoder fully connected layer parameters, respectively. i =( i ε , i D ) is the set of all parameters of the model.

[0124] 3. Network training:

[0125] (1) Data preprocessing: Preprocess the audio data, save the extracted features as npy files, and divide the preprocessed text into training set, validation set, and test set in a ratio of 8:1:1;

[0126] (2) Forward propagation: computing the model’s predicted output ;

[0127] (3) Calculate the loss function: Calculate the gradient of the loss with respect to the model parameters , and use the Adam optimizer to update the parameters and set appropriate hyperparameters (such as learning rate, etc.). The optimization algorithm formula is as follows:

[0128] ;

[0129] In the formula, represents the mean absolute error, N batch represents the number of samples in each batch, Indicates l The actual animation sequence parameters of the group training set, Indicates l The parameters of the animation sequence predicted by the group training set.

[0130] This part of the joint loss function mainly consists of two parts: L = L face + γL head ,in L face and L head The loss functions corresponding to facial expression and head posture respectively, c It is the weighting coefficient in multi-task learning (set to 0.75 in the present invention), which is used to balance the impact of each subtask on training.

[0131] 4) Backpropagation: Update the parameters using gradient descent or its variants (such as the Adam optimizer):

[0132] ;

[0133] In the formula, or represents the learning rate, Represents the gradient of the loss function with respect to the parameters.

[0134] (5) Iterative training: In the iterative process of multiple training cycles (epochs), the training set is used for training in each cycle.

[0135] (6) After each training cycle, the validation set should be used to evaluate the model performance. The number of training iterations should be determined based on the evaluation results to obtain the animation sequence prediction model. At the same time, the changes in training loss and validation loss should be closely monitored to ensure that the model is effectively prevented from overfitting.

[0136] After each training cycle, the validation set should be used to calculate the validation loss of the model performance (see the section on calculating the loss function in back propagation for the specific calculation method), and the changes in the training loss and validation loss should be closely monitored. If the validation loss increases and the training loss decreases, it indicates that the model has begun to overfit. This process mainly ensures the generalization ability of the model and effectively prevents the model from overfitting, thereby realizing the evaluation of model performance.

[0137] Step 5: Action synthesis and rendering;

[0138] By applying the generated controlRig parameters to the 3D model of the virtual character (rendering on the UE side is an existing technology, and the detailed steps will not be repeated), facial expression driving and head posture can be achieved. This process is usually combined with facial skeletal animation (Facial Rigging, FG) technology to ensure that the generated facial movements are natural, smooth and synchronized with the voice. In order to improve the expressiveness, more facial expression controls are introduced in addition to lip-driven, such as blinking animation and eyebrow micro-expressions, etc., to enhance the realism and emotional expression of the virtual character.

[0139] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store static information and dynamic information data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the steps in the above method embodiment are implemented.

[0140] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0141] In addition, the present invention also provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiment when executing the computer program.

[0142] In addition, the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiment are implemented.

[0143] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0144] The present invention is not limited to the structures which have been described above and shown in the drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for driving a virtual human using a voice signal, characterized in that: The method includes: The original audio signal data is resampled to obtain a speech signal, and the linear prediction coefficient is solved based on the dynamic frame division strategy and linear prediction analysis technology to obtain the speech feature sequence of the speech signal; The speech feature sequence is enhanced by using dynamic feature enhancement technology, an enhanced feature sequence is obtained based on the processing result, and the enhanced feature sequence is used as input to construct an animation sequence prediction model; The animation sequence parameters are obtained based on the output results of the animation sequence prediction model, and the animation sequence parameters are combined with the virtual human character model to perform facial action synthesis and rendering operations; The resampling of the original audio signal data to obtain a speech signal, and solving the linear prediction coefficient based on the dynamic frame strategy and the linear prediction analysis technology to obtain the speech feature sequence of the speech signal includes: The original audio signal is subjected to anti-aliasing filtering and difference processing using an open-source audio data tool to obtain a resampled speech signal. The frame length of the speech signal is determined by calculating the characteristic short-time energy based on a dynamic framing strategy. The speech signal is divided according to the frame length and the frame shift, and after the division is completed, the speech signal is subjected to windowing processing using the Hamming window technique to obtain a preprocessed speech signal; The linear relationship between the speech signal and the speech sample is estimated based on the linear prediction analysis technology, and the linear equations are solved using the minimum mean square error criterion to obtain the linear prediction coefficients as the speech feature sequence of the speech signal; The method of enhancing the speech feature sequence by using the dynamic feature enhancement technology, obtaining the enhanced feature sequence based on the processing result, and using the enhanced feature sequence as input to construct the animation sequence prediction model comprises: Based on the dynamic feature enhancement technology, the time series length and feature dimension of the speech feature sequence changing with time are analyzed, and the time series length, feature dimension and speech feature sequence are spliced ​​to obtain an enhanced feature sequence; Using a facial capture device to record speech collaborative multimodal data including audio and facial animation, and exporting facial data after fine-tuning the speech collaborative multimodal data; The enhanced feature sequence is used as the input layer to build a prediction architecture model, and the facial data is used as the training set to train the prediction architecture model network to obtain the animation sequence prediction model.

2. The method for driving a virtual human using a voice signal according to claim 1, characterized in that: The linear relationship between the speech signal and the speech sample is estimated based on the linear prediction analysis technology, and the linear equations are constructed and solved using the minimum mean square error criterion to obtain the linear prediction coefficients as the speech feature sequence of the speech signal, including: The speech signal is represented as a linear combination of samples, the prediction error term of the linear combination of samples is analyzed, and the linear relationship between the speech signal and the speech sample is estimated based on the prediction error term and the linear prediction analysis technology; Based on the linear relationship result, the autocorrelation function of the speech signal is obtained, and the linear equation group is constructed and solved by using the autocorrelation function and the minimum mean square error criterion, and the linear prediction coefficient is obtained according to the solution result; The obtained sets of linear prediction coefficients are combined as speech feature parameters of the speech signal, and a speech feature sequence is obtained according to the speech feature parameters.

3. The method for driving a virtual human using a voice signal according to claim 2, characterized in that: The expression of the autocorrelation function is: ； In the formula, represents the autocorrelation function, N Indicates the frame length. k Indicates the index of the frame, represents the speech signal, Indicates forward offset k The speech signal after the frame, P Indicates the order of the linear prediction model.

4. The method for driving a virtual human using a voice signal according to claim 3, characterized in that: The prediction architecture model is constructed by taking the enhanced feature sequence as the input layer, and the facial data is used as the training set to perform network training on the prediction architecture model to obtain the animation sequence prediction model, which includes: The enhanced feature sequence is set as the input layer for feature map generation, and an encoder is constructed based on several groups of convolutional layers. The encoder is used to extract local features to obtain a temporal feature sequence. A fully connected layer is used to map the temporal feature sequence to the animation sequence parameter space to generate a decoder, and the output layer structure is determined based on the decoder. The input layer, encoder, decoder and output layer are combined to obtain a prediction architecture model. After preprocessing the facial data, the data is divided into training set, validation set and test set according to the proportion. The training set is used to verify the prediction output of the prediction architecture model, and the gradient of the loss to the prediction architecture model parameters is calculated based on the prediction output results. Based on the gradient results, the optimizer is used to update the prediction architecture model parameters, and the optimization algorithm is used to update the hyperparameters. The performance of the updated prediction architecture model is evaluated based on the validation set. The number of training iterations is determined based on the evaluation results to obtain the animation sequence prediction model.

5. The method for driving a virtual human using a voice signal according to claim 4, characterized in that: The expression of the prediction architecture model is: ； In the formula, represents the parameters of the animation sequence predicted by the prediction architecture model, f represents the overall mapping function of the prediction architecture model, z represents the enhanced speech features of the input, θ represents the parameter set of the prediction architecture model, represents the encoder function, represents the decoder function, θ ε and θ D They represent the encoder convolutional layer parameters and the decoder fully connected layer parameters respectively.

6. The method for driving a virtual human using a voice signal according to claim 5, characterized in that: The expression of the optimization algorithm is: ； In the formula, represents the mean absolute error, N batch represents the number of samples in each batch, Indicates l The actual animation sequence parameters of the group training set, Indicates l The parameters of the animation sequence predicted by the group training set.

7. The method for driving a virtual human using a voice signal according to claim 1, characterized in that: The method of obtaining animation sequence parameters based on the output result of the animation sequence prediction model, and combining the animation sequence parameters with the virtual human role model to perform facial action synthesis and rendering operations includes: Obtain animation sequence parameters based on the output result of the animation sequence prediction model, apply the animation sequence parameters to the virtual human 3D character model, and render the virtual human 3D character model using rendering technology; Facial expression driving and head posture are realized based on the rendering results, and facial expressions are smoothed and voice operations are performed in combination with facial skeleton animation technology to realize voice signal driving of virtual people.

8. A system for driving a virtual person using a voice signal, characterized in that: The system includes: The speech feature sequence acquisition module is used to resample the original audio signal data to obtain the speech signal, and solve the linear prediction coefficient based on the dynamic frame division strategy and linear prediction analysis technology to obtain the speech feature sequence of the speech signal; A sequence prediction model building module is used to enhance the speech feature sequence using dynamic feature enhancement technology, obtain an enhanced feature sequence based on the processing result, and use the enhanced feature sequence as input to build an animation sequence prediction model; A virtual facial rendering and synthesis module is used to obtain animation sequence parameters based on the output results of the animation sequence prediction model, and combine the animation sequence parameters with the virtual human character model to perform facial action synthesis and rendering operations; The resampling of the original audio signal data to obtain a speech signal, and solving the linear prediction coefficient based on the dynamic frame strategy and the linear prediction analysis technology to obtain the speech feature sequence of the speech signal includes: The original audio signal is subjected to anti-aliasing filtering and difference processing using an open-source audio data tool to obtain a resampled speech signal. The frame length of the speech signal is determined by calculating the characteristic short-time energy based on a dynamic framing strategy. The speech signal is divided according to the frame length and the frame shift, and after the division is completed, the speech signal is subjected to windowing processing using the Hamming window technique to obtain a preprocessed speech signal; The linear relationship between the speech signal and the speech sample is estimated based on the linear prediction analysis technology, and the linear equations are solved using the minimum mean square error criterion to obtain the linear prediction coefficients as the speech feature sequence of the speech signal; The method of enhancing the speech feature sequence by using the dynamic feature enhancement technology, obtaining the enhanced feature sequence based on the processing result, and using the enhanced feature sequence as input to construct the animation sequence prediction model comprises: Based on the dynamic feature enhancement technology, the time series length and feature dimension of the speech feature sequence changing with time are analyzed, and the time series length, feature dimension and speech feature sequence are spliced ​​to obtain an enhanced feature sequence; Using a facial capture device to record speech collaborative multimodal data including audio and facial animation, and exporting facial data after fine-tuning the speech collaborative multimodal data; The enhanced feature sequence is used as the input layer to build a prediction architecture model, and the facial data is used as the training set to train the prediction architecture model network to obtain the animation sequence prediction model.

Images