A virtual population generation training method, system, medium and program product
By extracting voice features and emotional tags, dividing control parameters, and combining associated attention calculations and time sliding windows, the virtual verbal generation process is simplified, solving the problem of large amount and high complexity of verbal generation in the prior art, and realizing Efficient and accurate lip animation synthesis and fast customization of virtual human animations.
Patent Information
- Application Number
- CN202411290185.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-14
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-09-14
AI Technical Summary
The prior art When virtual people interact with users with voice, the calculation amount is large during the lip generation process, resulting in high computing power demand. It is difficult for the prior art to achieve efficient and accurate lip animation synthesis, especially in real-time interactive scenarios. .
By obtaining audio samples with target emotions tags, extracting voice features and determining initial control parameters, using the correlation attention calculation module to divide control parameters into target and correlation parameters, using the transformation function to generate lip generation model, simplifying the calculation process and reducing calculation Force requirements, combined with time sliding window and user customization process, optimize the coordination of lip shape and facial movements.
While maintaining the accuracy of lip shape, it reduces the complexity and calculation overhead of lip shape control parameters, improves the efficiency and accuracy of lip shape generation, and realizes high realistic and fast customization of virtual human animations.
Smart Images

Figure CN118820787B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a virtual population generation training method, system, medium and program product. Background Art
[0002] With the booming development of virtual reality and digital entertainment industries, virtual humans have been widely used. In scenarios such as virtual assistants, online education, news broadcasts, and game audio and video, their dynamic virtual images can provide a more vivid and friendly interactive experience.
[0003] At present, when a virtual human interacts with a user through voice, the traditional lip-sync generation method mainly generates lip-sync animation by extracting the voice features of the speech text, determining the driving parameters of the corresponding virtual human model, and using the driving parameters to drive the virtual human's facial model. However, due to the continuity requirements of the lip-sync animation, technical personnel are usually required to develop a prediction model to predict the voice features (driving parameters) in a short period of time based on the current voice features (driving parameters), which requires a lot of calculations.
[0004] When related technologies generate virtual human lip shapes in real time, they usually need to perform predictive calculations on a large amount of facial drive data (i.e., control parameters), which requires high computing power. Summary of the invention
[0005] The present application provides a virtual mouth shape generation training method, system, medium and program product, which are used to reduce the number of calculations of control parameters and simplify the mouth shape generation process to save computing power.
[0006] In the first aspect, the present application provides a virtual human mouth shape generation training method, which is applied to a training system, and the method includes: obtaining an audio test sample with a target emotion label, and extracting the speech features of each frame data of the audio test sample; determining the initial control parameters related to lip shape control in the character model of the virtual person; determining the control numerical matrix corresponding to the initial control parameters according to the feature comparison model of the speech features and the target emotion label; the control numerical matrix is composed of multiple control numerical vectors, and the control numerical vector includes the values of the initial control parameters at multiple time sampling nodes; inputting the control numerical matrix into an associated attention calculation module to obtain an associated index vector reflecting the degree of association between the control numerical vectors; according to the associated index vector, dividing the initial control parameters corresponding to the control numerical vector into target control parameters and associated control parameters; the associated control parameters are obtained by converting the corresponding target control parameters through a conversion function; taking audio training samples with different emotion labels as input and the target numerical matrix corresponding to the target control parameters as a supervision signal, performing model training to obtain a lip shape generation model; the lip shape generation model is bound to the emotion label and the corresponding conversion function.
[0007] In the above embodiment, the training system determines the initial control parameters related to lip control in the character model of the virtual person, and then inputs the corresponding control value matrix into the associated attention calculation module, determines the representative target control parameter among multiple interrelated related parameters according to the association index, excludes the associated control parameters associated with the target control parameters from the training process, and trains to obtain the lip generation model by focusing on the target control parameters. Through this method, the amount of calculation in the lip generation process can be simplified, and only a small amount of target control parameters need to be calculated. Other associated control parameters can be obtained from the target control parameters through the conversion function, and no large amount of computing resources are required for parameter prediction, thereby reducing the computing power requirements for lip generation and improving the efficiency of lip generation.
[0008] In combination with some embodiments of the first aspect, in some embodiments, the step of determining initial control parameters related to lip control in the character model of the virtual human specifically includes: determining preset control parameters related to lip control in the character model of the virtual human; removing control parameters that are not associated with speech features in the preset control parameters to obtain initial control parameters.
[0009] In the above embodiment, when determining the initial control parameters in the character model, the training system reduces the number of parameters that need to be processed for training by removing the control parameters that are irrelevant to the speech features and simplifying the lip shape control parameters to a set of initial control parameters that are only related to the speech features. The number of parameters required for lip shape control can be compressed, and the lip shape control task can be simplified to a prediction task of a smaller number of key parameters, which reduces the complexity of the model and improves the lip shape generation speed.
[0010] In combination with some embodiments of the first aspect, in some embodiments, the step of determining preset control parameters related to lip control in the character model of a virtual human specifically includes: determining the character model of the corresponding virtual human based on the user's option selection for the model feature parameters; determining the preset control parameters related to lip control based on the facial bone structure and muscle distribution of the character model.
[0011] In the above embodiment, the training system determines the character model of the corresponding virtual human according to the user's selection, and determines the preset control parameters related to lip control according to the facial bone structure and muscle distribution of the character model. The user can select a virtual human model with different characteristics according to needs, extract the preset control parameters related to lip control, and ensure the accuracy of lip animation generation through the matching of control parameters and character models.
[0012] In combination with some embodiments of the first aspect, in some embodiments, the steps of performing model training using audio training samples with different emotion labels as input and the target numerical matrix corresponding to the target control parameters as the supervision signal to obtain the lip shape generation model specifically include: using multiple audio samples with different emotion labels as training input and the target numerical matrix corresponding to the target control parameters as the supervision signal to perform model training of the lip shape generation model; during the training process, determining the adjustment vector of the model parameters according to the loss function between the model prediction output of the lip shape generation model and the target numerical matrix; when the performance index of the lip shape generation model on the verification set reaches a preset threshold, freezing the model parameters to obtain the final lip shape generation model.
[0013] In the above embodiment, the training system determines the adjustment vector of the model parameters according to the loss function between the model prediction output of the lip generation model and the target numerical matrix, and obtains the final lip generation model when the performance meets the standard. Through continuous iterative training and verification using the validation set, the model parameters are dynamically adjusted to ensure the efficiency and quality of the entire lip generation process.
[0014] In combination with some embodiments of the first aspect, in some embodiments, after the step of performing model training using audio training samples with different emotion labels as input and the target numerical matrix corresponding to the target control parameters as a supervisory signal to obtain a lip generation model, the method also includes: determining the overall control parameters related to facial control, head control and body control in the character model of the virtual human; performing model training using audio training samples with different emotion labels as input and the target numerical matrix corresponding to the target control parameters and the overall numerical matrix corresponding to the overall control parameters as supervisory signals to obtain an action generation model.
[0015] In the above embodiment, after obtaining the lip shape generation model, the training system will also determine the overall control parameters related to facial control, head control and body control in the virtual human role model, and perform model training to obtain the corresponding action generation model. Combining lip shape control with facial, head and body control achieves the effect of synchronously driving the virtual human mouth shape, facial expression, head movement and body movement, making the virtual human's action performance more natural and coordinated, and improving the human-computer interaction experience.
[0016] In combination with some embodiments of the first aspect, in some embodiments, after the step of performing model training with audio training samples with different emotion labels as input, and taking the target numerical matrix corresponding to the target control parameters and the overall numerical matrix corresponding to the overall control parameters as supervision signals to obtain the action generation model, the method also includes: determining the interaction control parameters related to the interaction with the environment in the character model; based on the interaction control parameters, correcting the action generation model to obtain the interaction generation model.
[0017] In the above embodiment, the training system determines the interaction control parameters related to the interaction with the environment in the role model, and determines the interaction generation model based on the interaction control parameters. On the basis of action generation, based on the interaction factors with the external environment, it can dynamically adjust the virtual person's response action according to the feedback information of the interactive environment (or object), forming multiple forms of interaction such as question-and-answer interaction and emotional interaction, making the behavior of the virtual person more intelligent and improving the user experience.
[0018] In combination with some embodiments of the first aspect, in some embodiments, the step of determining the control numerical matrix corresponding to the initial control parameters based on the feature comparison model of the speech features and the target emotion label specifically includes: determining the number of time sampling nodes and the sampling interval according to user settings; determining multiple time sliding windows corresponding to the multiple time sampling nodes; the time sliding window includes the time sampling node and a preset number of pre-sampling nodes; for each time sliding window, determining the control numerical matrix corresponding to the initial control parameters based on the feature comparison model of the speech features and the target emotion label.
[0019] In the above embodiment, the training system determines multiple time sliding windows corresponding to multiple time sampling nodes, and for each time sliding window, determines a control numerical matrix corresponding to the initial control parameters. Through the controllable time window, the generated control numerical matrix can be adjusted. The width change of the time window can be used to change the complexity of the control numerical matrix, providing the possibility of balancing the efficiency and accuracy of numerical calculations, and enhancing the adaptability of the model.
[0020] In a second aspect, an embodiment of the present application provides a training system, which includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the training system to perform the method described in the first aspect and any possible implementation method of the first aspect.
[0021] In a third aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when the computer program product is run on a training system, enables the training system to execute the method described in the first aspect and any possible implementation of the first aspect.
[0022] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions, which, when executed on a training system, enables the training system to execute the method described in the first aspect and any possible implementation of the first aspect.
[0023] It can be understood that the training system provided in the second aspect, the computer program product provided in the third aspect, and the computer storage medium provided in the fourth aspect are all used to execute the method provided in the embodiment of the present application. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method, which will not be repeated here.
[0024] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:
[0025] 1. By dividing the control parameters into target control parameters and associated control parameters, the associated control parameters are obtained by converting the corresponding target control parameters through a conversion function, and the corresponding model training method is adopted, thereby reducing the lip shape control parameters and model complexity while maintaining the lip shape accuracy, reducing the calculation overhead, and effectively solving the problems of redundant lip shape control parameters and high calculation complexity in the prior art, thereby achieving computing power saving.
[0026] 2. Due to the use of a sliding window mechanism in the time dimension and local speech feature extraction based on contextual information, as well as a method of constructing a control numerical matrix that time-aligns speech features, emotional features and lip parameters, it is possible to capture the dynamic changes in speech emotions, making the synthesized lip movements more compatible with the input speech in terms of emotional expression and time synchronization, effectively solving the problem of disconnection between synthesized lip shape and speech in the prior art, and thereby achieving improved accuracy in model-generated animation.
[0027] 3. Due to the adoption of a user-oriented virtual human customization process, the corresponding human body model is matched according to the appearance feature parameters selected by the user, and the coordination and synchronization of movements are optimized based on the overall control parameters of the lip shape, face, head and body. Therefore, the threshold and workload of virtual human design are lowered. Ordinary users do not need professional 3D modeling and animation production skills to flexibly customize personalized virtual humans according to interaction needs, which effectively solves the problems of high professionalism, long cycle and high cost in the construction of virtual humans in the existing technology, and thus realizes the rapid customization of virtual humans and high-fidelity voice-driven animation synthesis. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 This is a flow chart of a virtual population type generation training method in an embodiment of the present application;
[0029] Figure 2 This is another flowchart of the virtual population type generation training method in the embodiment of the present application;
[0030] Figure 3 It is a schematic diagram of a physical device structure of a training system in an embodiment of the present application. DETAILED DESCRIPTION
[0031] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification of the present application, the singular expressions "one", "a kind of", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to any or all possible combinations comprising one or more of the listed items.
[0032] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.
[0033] For ease of understanding, the application scenarios of the embodiments of the present application are introduced below.
[0034] Related studies have shown that facial movements of the human face are correlated in different areas, and the principle is derived from the physical constraints of coordinated muscle control and habitual patterns. Specifically, during stress and speech pauses, areas such as the mouth, eyebrows, and cheeks tend to move simultaneously to convey the intention of expression; in different emotional scenarios, facial associations have different rules. Based on this, in order to synthesize natural and realistic expressions, a method called CSTalk (Correlation Supervisored) is proposed in the related technology. This method models the correlation between facial movements and different areas of supervision, and supervises the training of the generative model to generate expressions that conform to human facial movement patterns; that is, through correlation constraints, the model face is made more realistic.
[0035] In related technologies, facial modeling uses the parametric Metahuman character model proposed by Epic, which operates facial animation through 185 control platforms. Each control platform corresponds to a set of facial muscles and has corresponding control parameters. By predicting the correct parameter change curve of the control platform from the voice, smooth facial animation can be controlled. Obviously, the real-time prediction of 185 complex data requires a high computing power.
[0036] However, in combination with the direction of this application, by modeling the correlation of 185 complex data, the data association can be identified. If there is data association, the interrelated data can be extracted, and one of the data can be assigned for prediction, and other related data can be directly calculated based on the data. This can achieve facial coordination changes while saving related computing power. For example, through association, 185 complex data can be reduced to 50 or even less.
[0037] For ease of understanding, the following describes the process of the method provided by this implementation in combination with the above scenario. Figure 1 , which is a flow chart of a virtual population type generation training method in an embodiment of the present application.
[0038] S101, obtaining an audio test sample with a target emotion label, and extracting speech features of each frame data of the audio test sample.
[0039] Among them, the audio test sample refers to the audio data used to test the effect of the lip generation model, including the speech content and the corresponding emotion label. The target emotion label indicates the emotion expressed by the audio segment, such as happy, sad, angry, etc. Speech features are used to represent the acoustic characteristics of audio signals. Common speech features include fundamental frequency, resonance peak, energy, etc. Frame data refers to a series of audio segments obtained by dividing the audio signal at fixed time intervals. Each segment is called a frame. For example, a 5-second audio segment can be divided into 500 frames of data with a frame length of 10 milliseconds.
[0040] First, the training system selects a batch of audios with the same target emotion label from the audio library as audio test samples, then performs frame-level segmentation on each audio test sample, extracts the speech feature vector of each frame, and obtains a complete feature sequence. The extracted speech features and the corresponding target emotion label will be used as input for subsequent steps to determine the corresponding control parameters for the Metahuman role model.
[0041] S102: Determine initial control parameters related to lip control in the character model of the virtual human.
[0042] The character model refers to a three-dimensional human model used to describe the appearance and behavioral characteristics of a virtual human, usually including bones, skin, hair and other components. Lip control refers to driving the movement of the mouth and surrounding areas by adjusting several parameters of the virtual human's face to match the input speech. The initial control parameters refer to all the parameters that are initially determined to directly control the virtual human's mouth shape.
[0043] Before generating a virtual human mouth sequence, it is necessary to determine a set of key parameters directly related to lip control as initial control parameters based on the character model to be used. Specifically, first load the pre-made virtual human character model, then analyze the facial area of the character model, identify the mouth and its surrounding bone structure and muscle distribution, and determine the degrees of freedom and constraints of each facial component. On this basis, select the initial control parameters that affect the lip movement from all the control parameters of the character model.
[0044] S103: Determine a control value matrix corresponding to initial control parameters according to a feature comparison model of speech features and target emotion labels.
[0045] Among them, the feature comparison model refers to the mapping relationship between speech features and facial parameters, which is usually trained from a large number of speech-lip data pairs using machine learning methods. The control value matrix refers to the matrix composed of the values of the initial control parameters at different time points. Each column corresponds to a time sampling point, and each row corresponds to a control parameter dimension containing all the initial control parameters. Each column can be regarded as a vector of parameter changes over time, that is, a control value vector.
[0046] After obtaining the speech feature sequence and target emotion label of the audio test sample, it is necessary to use the pre-trained feature comparison model to convert the speech features and emotion features into a lip control parameter sequence in order to determine the correlation between the subsequent multiple control parameters. Specifically, for each frame of the speech feature vector, it is spliced with the target emotion label as an input feature, and sent to the feature comparison model for forward reasoning to obtain the control numerical vector corresponding to the frame. Arrange the control parameter vectors of all frames in chronological order to obtain a complete control numerical matrix. The number of rows of the matrix is equal to the dimension of the initial control parameter, the number of columns is equal to the number of audio frames, and each element represents the value of a control parameter at a certain moment.
[0047] The expression using this control value matrix is:
[0048] y∈RT×R;
[0049] Among them, RT×R represents a matrix space with T rows and R columns (control numerical matrix), R is a positive integer, representing the control platform (corresponding to the initial control parameters), that is, the parameter vector used to control facial animation in the MetaHuman character model, and T is also a positive integer vector, representing the total number of frames of the animation; y is the output animation parameter curve (that is, the control numerical vector), that is, the curve fitted by time and control parameters.
[0050] In the calculation process of this control parameter curve (i.e., control numerical vector), the corresponding loss function is:
[0051] ;
[0052] Where N is the number of samples in the training data set, and They represent the value of the control platform curve and the predicted verification value respectively. The latter part represents the difference between adjacent curve frames. The smaller the numerical control, the smoother the parameter change. The loss function can be used to control the performance of the feature control model.
[0053] S104. Input the control value matrix into the correlation attention calculation module to obtain a correlation index vector reflecting the degree of correlation between the control value vectors.
[0054] Among them, the correlation attention calculation module refers to an implementation of an attention mechanism, which is used to calculate the correlation and dependency between control parameters. The correlation index vector represents the correlation strength between the control value vectors. The larger the value, the higher the correlation between the two control parameters, and they should change in conjunction.
[0055] After obtaining the numerical matrix of the initial control parameters, it is necessary to further analyze the intrinsic connections between these parameters and find the parameter subsets that affect each other in order to perform parameter compression. Specifically, the control numerical matrix is input into the correlation attention calculation module, and the correlation coefficients between the control parameters are calculated through the attention mechanism of the transformer architecture to obtain a correlation index matrix. Each element represents the correlation strength between the two control parameters in the corresponding row and column. Each row of the matrix is regarded as the correlation index vector of a control parameter to all other parameters, reflecting the correlation pattern of the control parameter with other parameters.
[0056] Referring to step S103, the control value matrix is RT×R;
[0057] The control value matrix is input into the associated attention calculation module, that is, the attention formula in the transformer architecture is calculated as follows:
[0058] ;
[0059] Among them, Q, K, and T act as attention weights, representing the influence between different positions in the sequence; Q and K represent two different control numerical vectors in the numerical matrix, Q is a certain vector, K is any other vector, V is the weighted vector preset by the model, and Softmax is the activation function used to normalize and arrange the relevant probabilities. is the scaling factor, and the final Attn(Q, K, V) is used to represent the correlation between Q and K.
[0060] Correspondingly, in the model calculation process of the associated attention calculation module, the correlation scores of other vectors in each layer of a certain control numerical vector are added to obtain the relevant feature representation of a certain control numerical vector, and its loss function is:
[0061] ;
[0062] Among them, M is the number of classifications, N is the number of samples in the training data set, represents the probability that the true label of the i-th sample is the c-th classification, is the predicted probability.
[0063] In some embodiments, the control value matrix can be input into the encoder of the transformer, and the attention score matrix between the control parameters can be calculated through the self-attention mechanism; the attention score matrix is row-normalized to obtain a correlation index matrix, and each row is a correlation index vector. Optionally, a graph convolutional neural network is used to process the control value matrix, and the control parameters are regarded as nodes of the graph, and the temporal correlation between frames is regarded as the edge of the graph. The neighborhood information of the nodes is aggregated through the graph convolution operation to obtain a node embedding vector containing correlation information as the correlation index vector. It can be understood that there are many ways to calculate the correlation index vector, such as Pearson correlation coefficient, mutual information, etc. The purpose of this step is to explore the correlation pattern between the control parameters and provide a basis for parameter screening. The specific implementation method is not limited.
[0064] S105 . According to the association index vector, the initial control parameters corresponding to the control numerical vector are divided into target control parameters and association control parameters.
[0065] Among them, the target control parameters refer to some key parameters selected from the initial control parameters, which have a decisive influence on the final lip shape. The associated control parameters refer to other parameters that are highly correlated with the target control parameters and can be calculated through the values of the target control parameters.
[0066] After obtaining the correlation index vector of each control parameter, it is necessary to divide the initial control parameters into two non-overlapping subsets according to the correlation strength: target control parameters and associated control parameters. Specifically, for the correlation index vector of each control parameter, select several elements with the largest values and mark the corresponding control parameters as strongly correlated parameters of the parameter. The number of times a parameter is marked as a strongly correlated parameter by multiple other parameters reflects the importance of the parameter. The control parameters with the largest number of marked times are selected as the target control parameters, and the parameters associated with the target control parameters are selected as the associated control parameters. The remaining parameters that have no correlation also need to be used as target control parameters, that is, the parameters that depend on other parameters are used as associated control parameters, and the remaining parameters with strong independence are used as target control parameters.
[0067] In some embodiments, the initial control parameters can be divided in a variety of ways: Optionally, the association index vector is truncated to Top-K, the K elements with the largest values are retained, and the remaining elements are set to zero. The truncated association index vector is summed up to obtain the association score of each parameter. The association scores are sorted from large to small, and the top L control parameters are taken as target control parameters. Optionally, an association graph of the control parameters is constructed based on the association index vector, where the nodes represent the control parameters and the edge weights represent the association index. Community division is performed on the association graph using a community discovery algorithm to obtain several parameter communities. The central node in each community is selected as the target control parameter, and the other nodes automatically become associated control parameters. It can be understood that there are many methods for extracting the target control parameters, such as clustering, submodule discovery, etc. The core of this step is to find a few important parameters to simplify the optimization space of the lip control task, and the specific implementation method is not limited.
[0068] For the relationship between the associated control parameters and the target control parameters, that is, the corresponding conversion function, two arrays of target parameters and associated parameters can be collected to form a paired data set, and a machine learning model can be trained on the data set to fit the mapping function from the target parameters to the associated parameters. Optionally, a general nonlinear regression model such as a multi-layer perceptron, radial basis function network, or a more complex generative network such as a variational autoencoder, a generative adversarial network, can be used to automatically extract the intrinsic correlation between the two sets of parameters from the data.
[0069] S106, taking the audio training samples with different emotion labels as input and the target numerical matrix corresponding to the target control parameters as the supervision signal, performing model training to obtain a lip shape generation model.
[0070] The audio training sample refers to a labeled data set used to train the lip generation model, which contains a large amount of paired speech-emotion-lip data. The target numerical matrix refers to a simplified control numerical matrix that only contains the target control parameter values. The complete lip parameters can be estimated by associating the control parameters.
[0071] According to the above steps, different target control parameters can be generated for different emotion labels. The principle is that there are different association rules for the changes in human faces corresponding to different emotions. After determining the target control parameters, the last step is to train the lip generation model so that it can automatically generate natural lip action sequences according to speech and emotions. Specifically, according to the preset annotated audio-lip dataset. The dataset is divided into a training set and a validation set. For each sample in the training set, the speech features and emotion labels are extracted as input, and the target numerical matrix corresponding to the target control parameters is extracted as the supervision signal, which is sent to the lip generation model for training. The output of the model is the predicted value of the target control parameter. The error is calculated with the real target numerical matrix to obtain the loss function value. Use optimization algorithms such as stochastic gradient descent to minimize the loss function and update the model parameters iteratively. Regularly evaluate the model performance on the validation set. When the preset convergence condition is reached, stop training to obtain the final lip generation model. The lip generation model can use the preset model to identify the target emotion according to the new speech input, and then generate realistic lip animation in real time, which is applied to virtual human interaction.
[0072] The overall loss function is:
[0073] , among which and See step S103 and step S104, which respectively represent the loss of control parameter prediction and the loss of correlation prediction.
[0074] In some embodiments, the lip generation model can be trained in a variety of ways: Optionally, a generation model based on a temporal convolutional network (TCN) is constructed, the input features are the concatenation of speech features and emotional features, and the output features are the numerical matrix of the target control parameters; the loss function uses the L1 norm, and the evaluation index is the prediction accuracy of the control parameters; the Adam optimizer is used to train 100 epochs at a learning rate of 0.001. Optionally, a sequence-to-sequence (Seq2Seq) model based on GRU is built, the encoder encodes the speech features into semantic vectors, and the decoder decodes the target control parameters at the current moment according to the semantic vectors and historical control parameters; the loss function uses the mean square error (MSE), and the evaluation index is the lip similarity; the RMSprop optimizer is used to train 50 epochs at a learning rate of 0.0001. It can be understood that there are many options for the structure and training methods of the lip generation model, such as generative models such as GAN and VAE, as well as different sequence modeling units and loss functions. The focus of this step is to obtain a simplified lip model that only depends on the target control parameters, and the specific implementation method is not limited.
[0075] The following is a more detailed description of the process of the method provided by this implementation. Figure 2, is another flow chart of the virtual population type generation training method in an embodiment of the present application.
[0076] S201, obtaining an audio test sample with a target emotion label, and extracting speech features of each frame data of the audio test sample.
[0077] Referring to step S101 , the training system extracts speech features of the audio test sample.
[0078] S202: Determine preset control parameters related to lip control in the character model of the virtual human.
[0079] Among them, preset control parameters refer to a set of facial control parameters pre-defined in the virtual human modeling stage, usually including deformation parameters of different areas such as jaw, mouth, cheek, etc.; these parameters control the facial shape of the character model and are the basis for lip shape generation.
[0080] Before extracting the lip control parameters, it is necessary to determine a set of preset control parameters related to lip expressions as a candidate feature set based on the specific structure of the virtual human character model. Specifically, the training system will call the production interface of the character model, obtain the facial structure and control method of the model, and determine the corresponding preset control parameters. For example, the Metahuman character model has 185 preset control parameters.
[0081] In some embodiments, the user can import the model into 3D animation software such as Maya, and use the BlendShape tool to create a set of deformation targets for mouth expressions, such as open mouth, closed mouth, pouting, etc.; each deformation target is connected to a controller, and the associated value of the controller is extracted as the preset control parameter. It is understandable that the acquisition of the preset control parameters needs to be combined with the specific 3D model file format and production process, and different methods such as code parsing or manual extraction can be used, as long as a set of key variables that affect the mouth shape can be found, which is not limited here.
[0082] In some embodiments, the training system determines the character model of the corresponding virtual person based on the user's option selection for the model feature parameters; and then determines the preset control parameters related to lip control based on the facial bone structure and muscle distribution of the character model.
[0083] Among them, model feature parameters refer to a set of high-level parameters that describe the appearance characteristics of the virtual human, such as age, gender, skin color, etc. Users can select or adjust the slider through the graphical interface to generate a customized virtual human image. The character model refers to the virtual human 3D model file corresponding to the model feature parameters, which contains information such as meshes, textures, bones, animations, etc. The facial bone structure refers to the bone topology of the virtual human face, which determines the deformation method of the skin. Muscle distribution refers to the muscle composition and influence area of the virtual human face, which is closely related to the generation of facial expressions.
[0084] Before generating the virtual human mouth animation, the appropriate virtual human character model will first be selected according to the user's preference. Specifically, the training system provides an interactive interface that displays virtual human thumbnails of different ages, genders, skin colors and other attributes. The user adjusts the attributes of interest by clicking or sliding the slider to preview the corresponding virtual human image in real time. After determining the final model feature parameters, the corresponding high-precision three-dimensional human body model is loaded from the model library as the character model for subsequent lip shape generation. After obtaining the character model, the training system automatically parses the bone hierarchy of the model, focusing on analyzing the bone topology of the facial area. At the same time, referring to anatomical knowledge, the distribution area of the main muscles is drawn on the facial surface. On this basis, the training system enumerates the facial preset parameters of the character model, combines the bone and muscle distribution, and selects a set of key control parameters related to lip shape expression for subsequent lip shape mapping learning.
[0085] In some embodiments, the role model and preset control parameters of the corresponding virtual person can be determined in a variety of ways: Optionally, an image-based modeling method is used to allow users to upload multiple personal photos, and a virtual head model similar to the photo is generated through a three-dimensional reconstruction algorithm; then a universal face skeleton template is used to bind the reconstructed model, and the deformation controller related to the mouth shape is extracted according to the semantic information of the skeleton as the preset control parameters. Optionally, a parameterized human body model, such as FaceWareHouse, FLAME, etc., is used to directly generate a corresponding standard face model according to the attributes such as age and gender input by the user; then a physics-based muscle simulation engine is used to solve the facial expression, and the degree of mouth muscle contraction is extracted as the preset control parameter. It can be understood that there are many ways to generate the virtual human role model and extract the mouth shape parameters, such as semantic segmentation, motion capture, expression migration, etc. The focus of this embodiment is to match user needs and provide a suitable face model and mouth shape control parameters, and the specific implementation method is not limited.
[0086] S203: Remove control parameters that are not related to speech features from the preset control parameters to obtain initial control parameters.
[0087] Among them, unrelated control parameters refer to control parameters that have little effect on lip shape changes or have no direct correspondence with speech pronunciation, such as parameters of facial areas such as eyes and nose.
[0088] After obtaining a set of preset lip control parameters, it is necessary to further analyze the correlation between each parameter and the speech features, eliminate redundant or irrelevant parameters, and obtain a more compact set of initial control parameters. Specifically, firstly, statistical analysis is performed on several annotated speech-lip data pairs to calculate the mutual information or correlation coefficient between each preset control parameter and the speech features. Then, the parameters with lower scores are automatically removed according to the correlation score sorting; or, based on prior knowledge, the parameters that are obviously irrelevant to the lip shape, such as eye rotation, are manually removed. The parameter subset finally obtained is used as the initial control parameter and will be used for subsequent lip shape generation control.
[0089] In some embodiments, control parameters that are not related to speech features in the preset control parameters can be removed in a variety of ways: Optionally, based on knowledge of facial anatomy, facial muscle control parameters outside the oral cavity, such as eyebrows, eyes, nose wings, etc., are manually excluded, and only muscle control parameters acting on the lips, tongue, and jaw are retained; using the production experience of animators, the lip shape parameters are manually adjusted to remove singular parameters that have no obvious effect on the lip shape or cause lip shape distortion. Optionally, a Wrapper feature selection algorithm is used to construct a simple linear regression model with speech features as input and lip shape parameters as output; through iterative training and parameter importance scoring, a subset of lip shape control parameters with the highest correlation with the input is automatically screened out as the initial control parameters.
[0090] S204: Determine a control value matrix corresponding to initial control parameters according to a feature comparison model of speech features and target emotion labels.
[0091] Referring to step S103 , the training system determines a control value matrix.
[0092] In some embodiments, the training system determines the number of time sampling nodes and the sampling interval according to user settings; then determines multiple time sliding windows corresponding to the multiple time sampling nodes; the time sliding window includes the time sampling node and a preset number of pre-sampling nodes; for each time sliding window, the training system determines the control numerical matrix corresponding to the initial control parameters according to the feature comparison model of the speech features and the target emotion label.
[0093] Among them, the time sampling node refers to the sequence of time points obtained by discretizing the continuous speech signal on the time axis, which is usually evenly distributed at fixed time intervals. The number of time sampling nodes and the sampling interval are flexibly set by the user according to needs, which determines the time resolution and precision of the lip sequence. The time sliding window refers to a time window centered on the current time sampling node and containing several preceding frames of data, which is used to extract the local speech-lip mapping relationship. The preceding sampling nodes refer to those sampling nodes in the time sliding window that are located before the current sampling node, providing context information.
[0094] In order to improve the time accuracy and synchronization of lip generation, it is necessary to perform more fine-grained time sampling of speech features and extract local speech-lip mapping relationships. Specifically, the training system first obtains the number of time sampling nodes and sampling intervals set by the user, such as 30 nodes per second and an interval of 33.33ms. Then, uniform sampling is performed on the speech signal to obtain a series of discrete time points. With each sampling node as the center, a fixed number (such as 10 frames) of speech frames are taken forward to form a time sliding window, ensuring that the window length is slightly larger than the duration of the average speech unit. In each sliding window, the acoustic features and corresponding emotional labels of the speech segment are extracted and sent to the pre-trained feature comparison model to obtain the lip control parameter values corresponding to this speech segment. The control parameter values of all sliding windows are arranged in chronological order, that is, a complete control parameter matrix is obtained, with columns representing the time dimension and rows representing the parameter dimension. This step uses a sliding window to align speech and lip shape, which can capture more subtle time changes and make the lip shape animation more accurate and smooth.
[0095] S205. Input the control value matrix into the correlation attention calculation module to obtain a correlation index vector reflecting the degree of correlation between the control value vectors.
[0096] Referring to step S104 , the training system determines a correlation index vector.
[0097] S206 . According to the association index vector, the initial control parameters corresponding to the control numerical vector are divided into target control parameters and association control parameters.
[0098] Referring to step S105 , the training system divides the initial control parameters into target control parameters and associated control parameters.
[0099] S207, using a plurality of audio samples with different emotion labels as training inputs, and using a target numerical matrix corresponding to the target control parameters as a supervisory signal, to perform model training of the lip generation model.
[0100] Among them, audio samples with different emotion labels refer to the annotation of emotional attributes added to the speech samples while recording the speech, such as happiness, sadness, anger, etc. The lip generation model refers to a machine learning model that takes a speech feature sequence as input and generates a lip parameter sequence synchronized with the speech, which is used for speech-driven lip animation synthesis.
[0101] Specifically, the speech data with emotion labels are divided into training set, validation set and test set. A batch of speech samples are randomly sampled from the training set to extract their speech features and emotional features. The feature data is input into the lip generation model to decode the predicted target control parameter matrix. The predicted matrix is then compared with the true target numerical matrix, the element-by-element error is calculated, and the overall loss function value is accumulated. The gradient backpropagation algorithm is used to calculate the gradient of the loss to the model parameters, and then the optimizer is used to update the model parameters to minimize the loss. After multiple rounds of iterative training, the lip generation model can learn the mapping relationship between speech-emotion-lip shape, which can be used to synthesize realistic lip animation in real time.
[0102] S208. During the training process, an adjustment vector of model parameters is determined according to a loss function between a model prediction output of the lip shape generation model and a target numerical matrix.
[0103] Among them, the model prediction output refers to the predicted value of a lip parameter matrix obtained by inputting a speech feature sequence into the current lip generation model. The target value matrix refers to the matrix composed of the actual lip parameter values corresponding to the speech features, which serves as a supervised learning signal. The loss function refers to a function that compares the difference between the predicted value and the actual value and is quantified by a scalar, which can be mean square error, cross entropy, KL divergence, etc. The adjustment vector refers to the direction and size of updating the current model parameters, which is determined by minimizing the loss function.
[0104] In each iteration of model training, it is necessary to evaluate the prediction performance of the current model and optimize the model parameters accordingly. Specifically, a batch of speech feature data is sampled from the training set, and the predicted output lip parameter matrix is obtained through the forward calculation of the current model. The predicted matrix is subtracted element by element from the true target numerical matrix, the error term is calculated, and then substituted into the loss function to obtain the loss function value of the batch. Then, the gradient descent method is used to calculate the partial derivative of the loss function for each model parameter to obtain the gradient vector. The gradient vector is multiplied by a small learning rate as the adjustment vector for this parameter update. The adjustment vector is subtracted from the current parameters to obtain the new model parameters for the next round of iteration. Through repeated iterations, the loss function will gradually decrease, and the prediction ability of the model will continue to improve until convergence. This step is the key to the automatic optimization of the machine learning model. The update direction and step size of the model parameters determine the final learned model performance.
[0105] For example, in the actual demonstration process, the present invention uses the live link face application on the iPhone to collect its own data, and uses the animation plug-in in Unreal Engine 5 to calculate the control platform data (185 preset control parameters). The data set includes five emotions: neutral, angry, sad, surprised and happy, and each emotion has 100 samples. The duration of each sample ranges from 4 seconds to 9 seconds. After excluding certain control devices that remain unchanged in all animations (control parameters that are not associated with voice features), only 116 (initial control parameters) are left, and each control value vector contains 96 frames of continuous data (frame rate is 30 fps), and the batch size and learning rate are set to 64 and 1e-5 respectively.
[0106] When training the lip generation model, this application uses the Librosa library to sample the speech audio at a rate of 16kHz, using a 96-frame time sliding window (corresponding to 51,200 audio samples) and a stride of 5 frames (sampling interval, corresponding to 2,667 audio samples). The processed audio data is input into a pre-trained Wav2vec 2.0 model, which includes 7 TCN layers, a projection layer, and 12 Transformer encoder layers, outputting a set of audio features a∈R159×116. The decoder then downsamples the features to 96 frames and passes them through the TCN layer. The final lip generation model is obtained.
[0107] S209: When the performance index of the lip shape generation model on the validation set reaches a preset threshold, the model parameters are frozen to obtain the final lip shape generation model.
[0108] The validation set refers to a portion of sample data set aside during model training, which is used to evaluate the generalization performance of the model and prevent overfitting. Performance indicators refer to quantitative measures used to measure the effectiveness of the lip generation model, and common ones include accuracy, error rate, precision, recall, etc. The preset threshold refers to the target value of a performance indicator that is pre-set according to specific application requirements. When the model performance reaches this value, the model is considered to have been trained. Freezing model parameters means stopping the update of model parameters, keeping the current parameters fixed, and saving the trained model to disk for later use.
[0109] During the model training process, the validation set data is used to evaluate the performance of the current model every certain number of training steps. Specifically, the speech feature data of the validation set is input into the model to obtain the predicted output lip parameter matrix. The predicted matrix is compared with the actual target numerical matrix to calculate performance indicators such as accuracy, average error, etc. The performance indicators of each round of verification are tracked and recorded. When the indicators of several consecutive rounds exceed the preset threshold and there is no obvious upward trend, it can be considered that the model has reached the optimal state and the training can be ended. At this time, stop the gradient update, freeze the model parameters, and save the trained model file for subsequent lip generation tasks. This step is to prevent the model from overfitting the training data, improve the generalization ability of the model, and avoid unnecessary waste of calculations.
[0110] In some embodiments, it is possible to determine whether the model has reached the optimal state in a variety of ways: Optionally, during the training process, the validation set loss function value of each epoch is recorded. When the loss value changes for 5 consecutive epochs are less than 0.001, the model is considered to have converged; during the training process, the validation set accuracy of each epoch is recorded. When the accuracy for 3 consecutive epochs is higher than 99%, the model is considered to have reached the optimal state; during the training process, the training time of each epoch is recorded. When the training time of each epoch exceeds 1 hour and the performance index has not been significantly improved, it is considered that the cost-effectiveness of continuing training is low and the training can be stopped. Optionally, an early stopping mechanism is used to track and record the optimal performance index of the validation set during the training process. When the performance index for 10 consecutive epochs does not exceed the current optimal value, the model is considered to have been overfitted and training is stopped; model compression techniques such as pruning, quantization, knowledge distillation, etc. are used to further compress the model volume after the model converges to obtain an equivalent lightweight model and accelerate the inference speed. It is understandable that determining whether the model has reached the optimal state requires combining specific task requirements and computing resources, and weighing model performance and training costs, which is not limited here.
[0111] S210: Determine overall control parameters related to facial control, head control, and body control in the character model of the virtual human.
[0112] Among them, facial control refers to controlling the movement of various organs on the face of the virtual person (such as eyes, eyebrows, nose, etc.) to express rich facial expressions. Head control refers to controlling the rotation of the virtual person's head and nodding, shaking, and other movements to match the rhythm of the voice. Body control refers to controlling the posture and movement of the virtual person's torso, limbs, and other parts to assist in expressing emotions. The overall control parameters related to face, head, and body control refer to a set of key parameters that affect the overall expressiveness of the virtual person's movements, such as emotional intensity, movement amplitude, time synchronization, etc., and coordinate the movements of the face, head, and body to match the emotions of the voice.
[0113] After completing the extraction and mapping learning of lip shape control parameters, the changes in lip shape alone are not enough to vividly express the emotions of the virtual person. In order to improve the overall expressiveness of the virtual person, it is also necessary to coordinate the movements of the face, head and body to match the rhythm and emotion of the voice. Specifically, the training system first defines a set of control parameters that affect the overall action effect of the virtual person, such as the degree of exaggeration of expression, frequency of nodding, gestures, etc. These parameters can be obtained through statistical analysis of a large amount of speech-action data to capture typical action patterns under different speech emotions. After obtaining the overall control parameters, they are input into the action generation model together with the extracted lip shape control parameters. The model explicitly or implicitly establishes the synergistic relationship between the lip shape, face, head and body in time and space, so that the movements of each part are synchronized in time and matched in emotion, forming a harmonious and unified overall action.
[0114] In some embodiments, the overall control parameters can be determined in a variety of ways: optionally, using a motion capture device to collect facial expressions and body movements during real-person dubbing, extracting the timing curves of each facial feature point and joint angle; performing frequency domain analysis on the timing curves, extracting the frequency bands with concentrated energy as the dominant frequency parameters of facial / body movements; and statistically calculating the maximum, minimum, mean, etc. of the timing curves as the amplitude parameters of facial / body movements. Optionally, using a rule-based expression coding system (such as FACS) and a posture coding system (such as BAP) to map speech emotions, obtain the distribution of typical emotions in facial action unit combinations and body posture categories; combining the emotional attributes of the lip control parameters, probabilistically sampling the facial expression action unit combination sequence and the body posture category sequence, and using the state transition probability of the sequence as the overall control parameter. It can be understood that the extraction of the overall control parameters can be based on demonstration data, expert knowledge, deep learning, and other methods, the purpose is to determine the correlation between the lip shape, face, head, and body in time and space, and guide the collaborative generation of action sequences, and the specific implementation method is not limited.
[0115] S211, taking audio training samples with different emotion labels as input, and taking the target numerical matrix corresponding to the target control parameters and the overall numerical matrix corresponding to the overall control parameters as supervision signals, perform model training to obtain an action generation model.
[0116] After obtaining the lip control parameters and overall control parameters, they need to be mapped to a complete human action sequence through machine learning. Specifically, the training system prepares a speech-action correspondence dataset with emotion labels, which contains multi-dimensional time series information such as lip shape, facial expression, head movement and body posture. With speech features as input and the time series matrix of lip target parameters and overall control parameters as output, an end-to-end action generation model is trained. This model can mine the statistical laws and time series dependencies of speech-emotion-action based on mainstream sequence modeling methods such as time convolution, recurrent neural network, and generative adversarial network. Through model training, the action generation model learns to synthesize virtual human animations that are synchronized with speech, emotionally matched, and coordinated with actions under the conditions of given speech input and target emotion. During testing, given a new speech clip, the action generation model can synthesize realistic virtual human expressions and actions in real time, and the lip shape is highly synchronized with the facial and body movements, and the overall emotional expression is natural and vivid.
[0117] In some embodiments, the action generation model can be trained in a variety of ways: optionally, a sequence-to-sequence model of an encoder-decoder structure is constructed, the encoder extracts high-level features of speech, and the decoder autoregressively generates the action control parameters of the current moment according to the speech features and historical action states; Dropout, L2 regularization and other methods are used to reduce the risk of overfitting, and batch normalization is used to accelerate the training process; Mixup, time expansion and other data enhancement strategies are used to expand the training set. Optionally, a spatiotemporal convolutional neural network is constructed, and the forward dependency of speech-action is modeled in the time dimension through causal convolution, and the correlation of lip shape, face, and body control parameters is modeled in the parameter dimension through hollow convolution; the network output simultaneously predicts future action parameters at multiple moments, and optimizes the smoothness of long-term motion through self-supervised loss function; the output action parameters are smoothed with Kalman filtering to improve the coherence of the action. It can be understood that the structure and training method of the action generation model are core research issues in the field of voice-driven animation, and a balance needs to be struck between expressive power and computational efficiency. The specific model architecture used in this embodiment is not limited.
[0118] In some embodiments, the training system determines the interaction control parameters related to the interaction with the environment in the character model; then based on the interaction control parameters, the action generation model is modified to obtain the interaction generation model.
[0119] Among them, environmental interaction refers to the interactive behavior between the virtual human and objects or other virtual humans in the surrounding environment, such as grabbing, pushing, pulling, collision, dialogue, etc. Interaction control parameters refer to a set of control parameters that affect the interactive ability of the virtual human, such as perception range, behavior decision tree, dialogue strategy, etc. The action generation model refers to a model that generates a complete action sequence of the virtual human based on voice and emotional input, and only considers the movement of the virtual human itself without involving environmental factors. The interaction generation model refers to an enhanced model formed by integrating the environmental interaction function on the basis of the action generation model, which can make real-time corrections to the original action based on environmental feedback.
[0120] Specifically, the training system first analyzes the virtual scene and extracts information about objects in the scene, such as location, size, material, function, etc. Then, a set of interaction control parameters is defined by combining the object information and the semantic information of the character, such as the operability of the object, the attention allocation of the character, and the interaction intention. The interaction control parameters and historical interaction records are input into the action generation model, and the model is incrementally trained through reinforcement learning and other methods, so that it can correct the original voice-driven actions according to the feedback information of the interaction object (such as displacement, force, etc.) and make more interactive actions that are more in line with the semantics of the environment. For example, when the hand touches the table, the fingers will automatically bend; when talking to people, the head will automatically turn to the speaker. The model after interaction enhancement is called the interaction generation model, which can control the virtual person to interact naturally with the surrounding environment through voice.
[0121] In some embodiments, the interaction control parameters and interaction generation model can be determined in a variety of ways: Optionally, a physical engine is used to perform collision detection on the virtual environment to obtain state information such as the spatial position of the object; a motion planning algorithm is used to calculate the motion trajectory of the key points of the human body so that it meets the environmental constraints; natural language processing technology is used to analyze the intention of the character dialogue, and the interaction type (such as greetings, questions, answers, etc.) is extracted as additional control parameters, and the dialogue is input into the action generation model to guide the style of the interactive action. Optionally, a graph neural network is used to model the object-character relationship graph in the scene, where the nodes in the graph are the key points of the object and the human body, and the edges are the spatial relationship between the two (such as distance, direction, etc.); a graph convolutional neural network is used to encode the node state, and the next moment position of the key points of the human body is output to realize the interaction between the character and the scene; a reinforcement learning framework is used to use the state of the graph encoding as the observation of the RL agent, and the character action as the action of the agent, and through trial and error learning, the character action is made to meet the scene constraints while maintaining voice synchronization. It is understandable that there are many methods and ways for virtual humans to interact with the environment, such as physical simulation, human-computer interaction, virtual reality, etc. The core of this embodiment is to give the originally independent voice-driven animation model the ability to perceive and adapt to the environment, and to enhance the interactive experience of the virtual human by integrating interactive intentions. The specific implementation method is not limited.
[0122] In the embodiments of the present application, a series of methods such as lip control parameter screening, temporal speech feature extraction, personalized virtual human customization, and global motion optimization are adopted to establish an end-to-end mapping framework from speech-emotion features to virtual human motion parameters. Therefore, the accuracy of lip synthesis and emotional expression are improved while the complexity of lip control is reduced. It also supports fast and flexible customization of personalized virtual human images, and generates highly realistic voice-driven animations with overall coordinated movements and lip synchronization. It effectively solves the deficiencies of existing voice-driven virtual human animations in terms of computing power requirements and authenticity, and thus realizes high-quality and personalized voice-driven real-time interaction of virtual humans.
[0123] The following describes the training system in the embodiment of the present invention from the perspective of hardware processing. Figure 3 , is a schematic diagram of a physical device structure of a training system in an embodiment of the present application.
[0124] It should be noted that Figure 3 The structure of the training system shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.
[0125] like Figure 3 As shown, the training system includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 302 or the program loaded from the storage part 308 to the random access memory (RAM) 303, such as executing the method described in the above embodiment. In RAM 303, various programs and data required for system operation are also stored. CPU 301, ROM 302 and RAM 303 are connected to each other through bus 304. Input / output (I / O) interface 305 is also connected to bus 304.
[0126] The following components are connected to the I / O interface 305: an input section 306 including an audio input device, a button switch, etc.; an output section 307 including a liquid crystal display (LCD) and an audio output device, an indicator light, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 310 as needed so that a computer program read therefrom is installed into the storage section 308 as needed.
[0127] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 309, and / or installed from a removable medium 311. When the computer program is executed by the central processing unit (CPU) 301, various functions defined in the present invention are performed.
[0128] It should be noted that specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.
[0129] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. Each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings.
[0130] Specifically, the training system of this embodiment includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, the virtual population type generation training method provided in the above embodiment is implemented.
[0131] As another aspect, the present invention further provides a computer-readable storage medium, which may be included in the training system described in the above embodiment; or may exist independently without being assembled into the training system. The above storage medium carries one or more computer programs, and when the above one or more computer programs are executed by a processor of the training system, the training system implements the virtual population type generation training method provided in the above embodiment.
[0132] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.
[0133] As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.
[0134] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.
Claims
1. A virtual population generation training method, characterized in that: Applied to a training system, the method comprises: Acquire an audio test sample with a target emotion label, and extract speech features of each frame data of the audio test sample; Determining initial control parameters related to lip control in the character model of the virtual human; Determine a control value matrix corresponding to the initial control parameter according to the speech feature, the target emotion label and the feature control model; the feature control model is used to determine the mapping relationship between the speech feature and the facial parameter, and the facial parameter is used to determine the control value vector; the control value matrix is composed of a plurality of control value vectors, and the control value vector includes the values of the initial control parameter at a plurality of time sampling nodes; Inputting the control value matrix into a correlation attention calculation module to obtain a correlation index vector reflecting the degree of correlation between the control value vectors; According to the association index vector, the initial control parameter corresponding to the control numerical vector is divided into a target control parameter and an associated control parameter; the associated control parameter is obtained by converting the corresponding target control parameter through a conversion function; The audio training samples with different emotion labels are used as input, and the target numerical matrix corresponding to the target control parameter is used as a supervision signal to perform model training to obtain a lip shape generation model; the lip shape generation model is bound with the emotion label and the corresponding conversion function.
2. The method according to claim 1, characterized in that The step of determining initial control parameters related to lip control in the character model of the virtual human specifically includes: Determining preset control parameters related to lip control in the character model of the virtual human; The control parameters that are not associated with the speech features are removed from the preset control parameters to obtain initial control parameters.
3. The method according to claim 2, characterized in that The step of determining preset control parameters related to lip control in the character model of the virtual human specifically includes: Determine the role model of the corresponding virtual person according to the user's option selection for the model feature parameter; According to the facial bone structure and muscle distribution of the character model, preset control parameters related to lip control are determined.
4. The method according to claim 1, characterized in that: The step of taking the audio training samples with different emotion labels as input and the target numerical matrix corresponding to the target control parameters as the supervision signal to perform model training to obtain the lip shape generation model specifically includes: Using a plurality of audio samples with different emotion labels as training inputs and a target numerical matrix corresponding to the target control parameters as a supervisory signal to perform model training of a lip generation model; During the training process, determining an adjustment vector of model parameters according to a loss function between a model prediction output of the lip generation model and the target numerical matrix; When the performance index of the lip shape generation model on the validation set reaches a preset threshold, the model parameters are frozen to obtain the final lip shape generation model.
5. The method according to claim 1, characterized in that After the step of taking the audio training samples with different emotion labels as input and the target numerical matrix corresponding to the target control parameters as the supervision signal to perform model training to obtain the lip shape generation model, the method further comprises: Determine overall control parameters related to facial control, head control, and body control in a virtual human character model; The audio training samples with different emotion labels are used as input, and the target numerical matrix corresponding to the target control parameters and the overall numerical matrix corresponding to the overall control parameters are used as supervision signals to perform model training and obtain an action generation model.
6. The method according to claim 5, characterized in that After the step of taking the audio training samples with different emotion labels as input, the target numerical matrix corresponding to the target control parameters, and the overall numerical matrix corresponding to the overall control parameters as supervisory signals, and performing model training to obtain the action generation model, the method further includes: Determine the interaction control parameters related to the interaction with the environment in the character model; Based on the interaction control parameters, the action generation model is modified to obtain an interaction generation model.
7. The method according to claim 1, characterized in that The step of determining the control value matrix corresponding to the initial control parameter according to the feature comparison model of the speech feature and the target emotion label specifically includes: Determine the number of time sampling nodes and sampling interval according to user settings; Determine a plurality of time sliding windows corresponding to a plurality of time sampling nodes; the time sliding windows include the time sampling nodes and a preset number of preceding sampling nodes; For each of the time sliding windows, a control value matrix corresponding to the initial control parameters is determined according to a feature comparison model of the speech features and the target emotion label.
8. A training system, characterized in that: The training system includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the training system to execute the method described in any one of claims 1-7.
9. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on a training system, the training system is caused to execute the method as claimed in any one of claims 1 to 7.
10. A computer program product, characterized in that When the computer program product is run on a training system, the training system is caused to perform the method according to any one of claims 1 to 7.
