Controller parameter generation method and device based on artificial intelligence and storage medium

Abstract

The invention discloses a controller parameter generation method and device based on artificial intelligence and a storage medium. The method comprises the following steps: acquiring voice data and emotion data corresponding to the voice data, extracting voice features from the voice data, and extracting emotion features from the emotion data; fusing the voice features and the emotion features based on a factor graph to obtain fused features; and performing animation parameter mapping processing on the fusion features to obtain controller parameters for driving facial expressions and mouth shapes of the virtual object. The technical problem that the facial expression and mouth shape of the virtual object are not natural is solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to an artificial intelligence-based controller parameter generation method, device and storage medium. Background Art

[0002] In the prior art, the facial expressions and lip shapes of virtual objects are usually driven only by voice data, ignoring the emotional information contained in the voice, resulting in the generated virtual object expressions and lip shapes appearing mechanical and lacking natural emotional expression, which affects the user's interactive experience. Especially in application scenarios where virtual objects need to communicate more emotionally with users, such as virtual assistants, game characters, animation production, etc., traditional methods are difficult to achieve delicate expressions that match the voice emotions. Therefore, how to effectively combine voice and emotional information to generate natural facial expressions and lip shape control parameters has become a problem that needs to be solved urgently.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present invention provide a controller parameter generation method, device and storage medium based on artificial intelligence to at least solve the technical problem of unnatural facial expressions and lip shapes of virtual objects.

[0005] According to one aspect of an embodiment of the present invention, there is provided a controller parameter generation method based on artificial intelligence, comprising: acquiring voice data and emotion data corresponding to the voice data, extracting voice features from the voice data, and extracting emotion features from the emotion data; fusing the voice features and the emotion features based on a factor graph to obtain fused features; and performing animation parameter mapping processing on the fused features to obtain controller parameters for driving facial expressions and lip shapes of virtual objects.

[0006] According to another aspect of an embodiment of the present invention, a method for driving a virtual object is also provided, comprising: using the controller parameters obtained by the above-mentioned controller parameter generation method based on artificial intelligence to drive the facial expression and lip shape of the virtual object.

[0007] According to another aspect of an embodiment of the present invention, there is also provided a controller parameter generation device based on artificial intelligence, including: an extraction module, configured to obtain voice data and emotion data corresponding to the voice data, extract voice features from the voice data, and extract emotion features from the emotion data; a fusion module, configured to fuse the voice features and the emotion features based on a factor graph to obtain fused features; and a data processing module, configured to perform animation parameter mapping processing on the fused features to obtain controller parameters for driving facial expressions and lip shapes of virtual objects.

[0008] In an embodiment of the present invention, voice data and emotion data corresponding to the voice data are obtained, voice features are extracted from the voice data, and emotion features are extracted from the emotion data; the voice features and emotion features are fused based on a factor graph to obtain fused features; animation parameter mapping is performed on the fused features to obtain controller parameters for driving the facial expressions and lip shapes of virtual objects. The above solution solves the technical problem of unnatural facial expressions and lip shapes of virtual objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0010] Figure 1 is a flow chart of a controller parameter generation method based on artificial intelligence according to an embodiment of the present application;

[0011] Figure 2 is a flow chart of another controller parameter generation method based on artificial intelligence according to an embodiment of the present application;

[0012] Figure 3 is a flow chart of extracting speech features according to an embodiment of the present application;

[0013] Figure 4 is a flow chart of extracting emotion features according to an embodiment of the present application;

[0014] Figure 5 is a flow chart of a method for performing fusion processing according to an embodiment of the present application;

[0015] Figure 6 is a flow chart of a method for obtaining controller parameters according to an embodiment of the present application;

[0016] Figure 7 is a structural schematic diagram of a controller parameter generation device based on artificial intelligence according to an embodiment of the present application;

[0017] Figure 8 A schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0018] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0019] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0020] According to an embodiment of the present invention, a method embodiment of a controller parameter generation method based on artificial intelligence is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0021] Figure 1 is a controller parameter generation method based on artificial intelligence according to an embodiment of the present invention, such as Figure 1 As shown, the method comprises the following steps:

[0022] Step S102, acquiring voice data and emotion data corresponding to the voice data, extracting voice features from the voice data, and extracting emotion features from the emotion data.

[0023] First, voice data and emotion data corresponding to the voice data are obtained. Voice data can be obtained through a microphone, a voice file or other audio input device, and can include multiple voice inputs. Emotion data can be manually input or automatically generated by an emotion recognition module, for example, by automatically identifying the emotional state by analyzing parameters such as the tone and speech speed of the voice.

[0024] Next, speech features are extracted from the speech data. For example, the speech data is subjected to preliminary signal processing, and original speech frames are extracted from the processed speech data; the original speech frames are subjected to convolutional neural network processing to obtain the speech features. Specifically, in some embodiments, the speech data is preprocessed, including operations such as noise reduction and speech signal enhancement, to improve signal quality. The original speech frames are extracted from the processed speech data, and signal processing techniques such as fast Fourier transform (FFT) or Mel-frequency cepstral coefficients (MFCC) can be used in the extraction process. Then, the original speech frames are input into a convolutional neural network (CNN) for feature extraction to obtain high-level speech features. Speech features can characterize important information such as the frequency, pitch, and rhythm of speech for subsequent processing.

[0025] Finally, the emotional features are extracted from the emotional data. For example, semantic analysis is performed on the emotional data to generate an emotional intensity matrix; the emotional intensity matrix is ​​input into an emotional recognition network to extract corresponding emotional features, and the extracted emotional features are normalized to generate standardized emotional features. Specifically, in some embodiments, the semantic analysis process can identify the semantic content in the speech through natural language processing (NLP) technology to infer the emotional state of the speaker. The emotional intensity matrix is ​​input into the emotional recognition network to extract corresponding emotional features, and the emotional features can cover the intensity of different emotions, such as "joy", "anger", "sorrow", "surprise", "fear", "disgust", etc. The extracted emotional features are normalized to ensure that different emotional features are comparable, and the standardized emotional features are generated so as to be fused with subsequent speech features.

[0026] Step S104: fusing the speech feature and the emotion feature based on the factor graph to obtain a fused feature.

[0027] For example, the factor graph is used to perform joint probability inference on the speech features and the emotional features to generate a fusion feature matrix; the fusion feature matrix is ​​subjected to dimensionality reduction processing to obtain the fusion features after dimensionality reduction. The factor graph model is a probabilistic graph model used to represent the dependency relationship between multiple variables. It can reduce complexity and enhance the correlation analysis between features through factor decomposition. Since the fusion feature matrix has a high dimension, dimensionality reduction helps to remove redundant information and retain the most important features, thereby reducing the model calculation complexity and overfitting risk. Through dimensionality reduction technology, more streamlined and optimized fusion features can be obtained to ensure efficient and precise control in subsequent animation generation.

[0028] In some embodiments, the dimensionality reduction process may be performed using the following method:

[0029] 1) Obtain a feature matrix to be reduced in dimension, and use a Gaussian kernel function to calculate a first similarity between each data point in the feature matrix in a first space.

[0030] For example, the first similarity is calculated based on the local density weight, the number of Gaussian kernel functions, the weight of the Gaussian kernel function, and the bandwidth parameter. The specific calculation formula is:

[0031]

[0032] in,

[0033] Among them, γ i is the local density weight of the ith data point, reflecting the data density of the area where the point is located, m is the total number of data points in the data set, σ is the parameter that controls the distance scaling between two data points, and is used to adjust the sensitivity of the density weight, xi represents the ith data point, xj represents the jth data point, P is the number of Gaussian kernel functions used, that is, the number of kernel function types, x i ′ is the i-th data point after mapping, x′ j is the jth data point after mapping.

[0034] 2) randomly mapping each of the plurality of data points corresponding to the feature matrix in the first space to a second space, wherein the dimension of the first space is greater than the dimension of the second space;

[0035] 3) Calculate the second similarity between each data point in the feature matrix in the second space using T distribution.

[0036] The second similarity can be calculated based on the Euclidean distance between two data points and the normalization factor of the similarity probability for all k not equal to i. The specific formula is:

[0037]

[0038] Where P(i|j) represents the similarity probability of data point xi′ to data point xj′ in low-dimensional space. D(i,j) represents the square of the Euclidean distance between data points xi′ and xj′ in low-dimensional space. The formula in the denominator represents the normalization factor for calculating the similarity probability for all k not equal to i, ensuring that the sum of all P(i|j) is 1.

[0039] 4) Based on the first similarity and the second similarity, adjusting the position of each data point in the second space to reduce the dimension of the feature matrix.

[0040] For example, for each pair of corresponding data points in the first similarity matrix and the second similarity matrix, the difference between the data point pairs is calculated to obtain a difference matrix; based on the difference matrix, the difference between the data point pairs is quantified using a divergence function to generate a divergence value; based on the divergence value, the divergence matrix is ​​generated, wherein the divergence matrix is ​​used to represent the position change of each data point in the first space and the second space. Afterwards, the divergence matrix is ​​used to construct an objective function representing the difference between the first similarity and the second similarity; by minimizing the objective function, the position of each data point in the second space is adjusted to obtain the adjusted data point position as the feature matrix after dimensionality reduction.

[0041] Among them, the objective function is:

[0042] Wherein, Ω(X′) is a regularization term, which is used to constrain the distribution of data points in the low-dimensional space to ensure that the distribution of data points is more uniform or meets specific application requirements. The regularization term can be, for example, the L2 norm, which is used to control the amplitude of the data point position.

[0043] Step S106, performing animation parameter mapping processing on the fusion feature to obtain controller parameters for driving the facial expression and mouth shape of the virtual object.

[0044] First, the fusion features after dimension reduction are input into a feedforward neural network. The feedforward neural network performs feature mapping on the fusion features based on a self-attention mechanism. The self-attention mechanism can capture key information by calculating the correlation between features, thereby improving the accuracy of animation generation.

[0045] Next, multi-head attention processing is performed on the feature mapping results. Multi-head attention processing uses multiple attention heads to simultaneously analyze the fusion features from different angles to capture information of different dimensions and generate preliminary animation parameters. These parameters include facial expressions, lip shapes, and other details of the virtual object that match the speech and emotional features.

[0046] Finally, a multi-frame time series analysis is performed on the preliminary animation parameters. The purpose of multi-frame time series analysis is to smooth the animation parameters by analyzing the dynamic changes between consecutive frames, thereby ensuring that the expressions and movements of the virtual objects are smooth and natural. The smoothed controller parameters can be used to drive the facial expressions, mouth shapes and postures of the virtual objects in real time, so that the performance of the virtual objects is highly matched with the speech content and emotional data. This precise animation generation can be widely used in scenarios such as virtual human live broadcasts and game character animations.

[0047] Figure 2 is a controller parameter generation method based on artificial intelligence according to an embodiment of the present invention, such as Figure 2 As shown, the method comprises the following steps:

[0048] Step S202: Acquire voice data and emotion data corresponding to the voice data.

[0049] First, voice data is obtained from input devices such as microphones, audio files, or other audio capture devices. Voice data can be real-time input voice signals or pre-recorded audio files. The input of these voice data is mainly used for subsequent feature extraction and sentiment analysis.

[0050] In order to further enhance the naturalness of the interaction between voice and virtual objects, emotional data corresponding to the voice data is obtained at the same time. Emotional data can be obtained in two ways: one is that the user manually enters the emotion label, and the other is through the automatic emotion recognition module, which automatically detects the speaker's emotional state using voice features (such as tone, speed and pitch, etc.). The automatic emotion recognition module can rely on a pre-trained emotion recognition model, which is trained with a large amount of speech data with emotion labels and can recognize a variety of emotions, such as joy, anger, sadness, etc.

[0051] Step S204: extracting speech features from the speech data.

[0052] Perform feature extraction on speech data. The specific implementation is as follows Figure 3 The following steps are shown:

[0053] Step S2042, performing preliminary signal processing on the voice data.

[0054] Initial signal processing usually includes operations such as noise reduction, dereverberation, and normalization. Noise reduction can remove background noise through techniques such as frequency domain filtering, dereverberation can eliminate echo effects in speech signals, and normalization ensures that speech signals of different volumes are processed within the same range.

[0055] Step S2044, extracting the original speech frame.

[0056] After completing the preliminary signal processing, the processed speech signal is divided into multiple short-term speech frames, each of which represents the speech signal within a certain period of time (usually 20ms to 50ms). The speech frames are extracted by sliding windows. The window length determines the time span of each frame of speech. There is a certain overlap between windows to ensure a smooth transition between frames.

[0057] Step S2046, convolutional neural network processing.

[0058] After extracting the speech frames, each speech frame is input into a convolutional neural network (CNN) for feature extraction. CNN can capture the timing patterns and frequency distribution characteristics in speech signals through a combination of multiple convolutional layers, pooling layers, and activation functions. The convolutional layer extracts local features in the speech frame, while the pooling layer is responsible for downsampling, reducing the size of the feature matrix while retaining important speech information. The final fully connected layer integrates the extracted local features into high-level speech features, which reflect the phonemes, rhythms, and other information of the speech, and are suitable for further sentiment analysis and animation driving.

[0059] Step S206: extracting emotion features from the emotion data.

[0060] Process the emotional data to extract emotional features. The specific methods are as follows: Figure 4 As shown, it includes the following steps:

[0061] Step S2062, performing semantic analysis on the emotion data.

[0062] The processing of emotional data is first completed through the semantic analysis module. This module analyzes the semantic content of emotional data through natural language processing (NLP) technology or speech emotion recognition technology. Specifically, language models such as word embedding (Word2Vec) and sentence embedding (BERT) can be used, or sound features such as pitch, speaking speed, and pauses can be combined to extract emotional clues in speech. After the analysis is completed, an emotional intensity matrix is ​​generated, which contains multiple emotion types and their corresponding intensity values ​​(such as "joy", "anger", "sadness", "surprise", etc.).

[0063] Step S2064, inputting the emotion intensity matrix into the emotion recognition network.

[0064] After the emotion intensity matrix is ​​generated, it is input into the emotion recognition network. The emotion recognition network is a deep neural network (such as RNN, LSTM or Transformer model), which is used to further extract high-level emotion features based on the emotion intensity matrix. The emotion recognition network can recognize and represent complex emotions, such as mixed emotions (joy mixed with surprise) or gradually changing emotions.

[0065] Step S2066, normalization processing.

[0066] In order to ensure the consistency of the comparison between different emotional features, the extracted emotional features are normalized. The role of normalization is to unify the numerical range of emotional features to facilitate subsequent fusion and processing. After normalization, standardized emotional features are generated, which can represent both the intensity of emotions and the relationship between different emotions.

[0067] Step S208: fusing the speech feature and the emotion feature based on the factor graph to obtain a fused feature.

[0068] The extracted speech features and emotional features are fused through the factor graph model to generate fused features suitable for driving the expression and mouth shape of the virtual object. Figure 5 As shown, the method comprises the following steps:

[0069] Step S2082, joint probability inference.

[0070] First, the extracted speech features and emotion features are jointly probabilistically inferred through a factor graph model. A factor graph is a flexible graphical model suitable for processing multidimensional data with complex dependencies. In the present invention, a factor graph model is used to describe the interaction between speech features and emotion features, and the probabilistic dependencies of these features are represented by nodes and edges in the graph. Specifically, the following steps are included:

[0071] 1) Voice features and emotion features are used as random variables.

[0072] In the factor graph, speech features and emotion features are treated as random variables. The nodes in the factor graph define the state of each feature, and the edges represent the mutual influence between these features. For example, features such as pitch and rhythm in speech affect the recognition of emotions, and the intensity of emotions may also affect the parsing of speech features.

[0073] 2) Probabilistic inference.

[0074] When performing joint probability inference, the factor graph jointly models the relationship between speech features and emotional features and calculates the conditional probability distribution of each feature, which can capture the subtle mutual influence between different features and ensure that speech and emotional information can be efficiently fused. Through these joint probabilities, the system can derive the fusion probability results of each group of features, thereby generating a fusion feature matrix.

[0075] 3) Generation of fusion feature matrix.

[0076] Through joint probability inference, speech features and emotional features are integrated into a matrix. The fused feature matrix not only contains the timing and frequency information of speech features, but also reflects the intensity and change trend of emotional features. The fused feature matrix provides a more complete data basis for the subsequent expression and lip animation generation.

[0077] Step S2084, dimensionality reduction processing.

[0078] The dimensionality reduction method in this embodiment is different from the dimensionality reduction method in the previous embodiment. In this embodiment, the high-dimensional fusion feature matrix generated by the fusion of the factor graph model has a large data dimension. In order to simplify the structure of the matrix while retaining the most important feature information, this application uses the t-SNE algorithm for dimensionality reduction. t-SNE can effectively retain the local structure of the data, is particularly suitable for processing high-dimensional data, and ensures that similar data points in the feature space still maintain adjacent relationships after dimensionality reduction. Specifically, the following steps are included:

[0079] 1) Initialization.

[0080] In order to start dimensionality reduction, first initialize the parameters of the t-SNE algorithm. t-SNE processes the local relationship between data points through the similarity between dimensions, so its core parameters need to be set appropriately to ensure that the dimensionality reduction results can reflect the local structure of the original data. The parameter Perplexity determines the size of the local neighborhood of each data point in the t-SNE model, and this embodiment is set between 5 and 50. Smaller perplexity will focus on local structures in a smaller range, while larger perplexity will focus more on the global data. The parameter Learning Rate is the learning rate, which determines the rate at which the model updates the embedded position of data points during dimensionality reduction. A learning rate that is too high will cause the data points to be scattered after dimensionality reduction, while a learning rate that is too low may cause the training process to be slow. Since t-SNE uses stochastic gradient descent to optimize the dimensionality reduction results, the number of iterations determines the degree of convergence of the optimization. t-SNE requires hundreds to thousands of iterations to ensure that the results are stable. In the initialization phase, an initial position is randomly assigned to each data point in a low-dimensional space (usually 2D or 3D space). Then, t-SNE optimizes these low-dimensional embedding positions so that the distance relationship between the data points after dimensionality reduction is as consistent as possible with the relationship in the high-dimensional space.

[0081] 2) Calculate the similarity between high-dimensional data points.

[0082] In the dimensionality reduction process of t-SNE, the similarity between each data point in the original high-dimensional fusion feature matrix needs to be calculated first. The similarity is calculated based on Gaussian distribution. For a given high-dimensional data point, t-SNE uses the Gaussian kernel function to calculate the similarity between the point and its neighboring points. The similarity is measured by the distance in the high-dimensional space. The closer the distance, the higher the similarity. Then, the local similarity matrix of each data point is normalized so that the sum of the similarities of each point is 1, which ensures that the local structure of each data point can be evenly considered during the dimensionality reduction process.

[0083] 3) Similarity adjustment in low-dimensional space.

[0084] Next, t-SNE calculates the similarity of each data point in the low-dimensional space using a Student's t distribution instead of a Gaussian distribution. The t distribution has a longer tail than the Gaussian distribution, so that the distances between data points in the low-dimensional space better reflect the local structure in the high-dimensional space.

[0085] 4) Initial random distribution of low-dimensional data points.

[0086] Each high-dimensional data point is initially randomly mapped to a low-dimensional space, where the distance between data points in the low-dimensional space reflects the similarity relationship in the high-dimensional space through iterative optimization.

[0087] 5) Gradient descent optimization.

[0088] Through stochastic gradient descent (SGD), the positions of data points in low-dimensional space are continuously adjusted to minimize the difference between the similarities of data points in high-dimensional and low-dimensional space. For example, divergence can be used as a loss function. At each iteration, the system adjusts the positions of data points so that similar data points in high-dimensional space are kept as close as possible in low-dimensional space, while dissimilar data points are kept away.

[0089] 6) Generate the fusion feature matrix after dimension reduction.

[0090] After multiple iterations, a stable fusion feature matrix after dimensionality reduction is finally generated in the low-dimensional space. Compared with the original high-dimensional fusion feature matrix, the dimension of this matrix is ​​significantly reduced, but it still retains the local similarity between speech and emotional features, ensuring that similar features are arranged closely in the low-dimensional space. This application ensures that the relationship between local neighborhoods in the original high-dimensional data is maintained in the low-dimensional space. After dimensionality reduction, similar speech emotion features can still reflect their correlation, thereby ensuring that the expression and lip shape changes of virtual objects are natural and coherent.

[0091] In some other embodiments, the following method may be used to perform dimensionality reduction:

[0092] Step S210, performing animation parameter mapping processing on the fusion feature to obtain controller parameters for driving the facial expression and mouth shape of the virtual object.

[0093] like Figure 6 As shown, the method for obtaining controller parameters includes the following steps:

[0094] Step S2102, inputting the fusion features into a feedforward neural network.

[0095] First, the system obtains the fusion feature matrix generated in the previous step. The fusion feature matrix combines speech features and emotional features, contains key information about speech content and emotional state, and is provided as input data to the feedforward neural network for processing.

[0096] The feedforward neural network is a multi-layer perceptron architecture used for feature mapping. Information is transmitted from the input layer to the output layer through several hidden layers. The output of each layer of nodes is only affected by the excitation of the previous layer of nodes. Information is transmitted in one direction and no loop occurs. Among them, the input layer receives the fused feature matrix, the dimension of which is the feature vector after dimensionality reduction. The hidden layer consists of several fully connected neurons, each of which performs a linear weighted sum and performs a nonlinear transformation through an activation function (such as ReLU, Sigmoid or Tanh). The number of hidden layers and the number of neurons in each layer can be dynamically adjusted according to the complexity of the task. The output layer provides the data results after feature mapping, usually the mapped feature vector, for subsequent processing.

[0097] Step S2104, feature mapping is performed based on the self-attention mechanism.

[0098] After obtaining the initial output of the feedforward neural network, the system introduces the self-attention mechanism to further optimize the feature mapping. The introduction of the self-attention mechanism enables the system to more flexibly and accurately capture the correlation between different dimensions in the fusion feature, thereby generating features with global relevance. The self-attention mechanism is a mechanism that can assign different attention weights to each element in the sequence (such as each feature dimension in the fusion feature), which can measure the contribution of each part of the fusion feature to other parts, and generate high-quality mapping results by paying attention to the dependencies between the parts.

[0099] Specifically, first, calculate the Query, Key, and Value vectors. The output feature vector of the feedforward neural network is transformed linearly to generate three vectors: Query, Key, and Value. For each input feature, Query is used to compare with other features, Key represents the information of other features, and Value is the information value that needs to be paid attention to. Next, calculate the attention weight. By performing a dot product operation on Query and Key, the similarity between each feature is measured, and the dot product result is normalized to the attention weight through the Softmax function. Then, perform a weighted sum. The Value vector of each feature is weighted and summed according to its corresponding attention weight to generate a new mapping feature. This feature mapping method can capture global contextual relationships and allow each feature to pay attention to other parts of the entire feature sequence, thereby better reflecting the complex interaction of speech and emotional features.

[0100] Through the self-attention mechanism, the system generates a new feature representation that not only contains the information of the original fused features, but also captures the deep dependencies between different features through the global attention relationship. The above processing greatly improves the accuracy of the feature representation, making the subsequent animation parameter generation more natural and coherent.

[0101] Step S2106, perform multi-head attention processing on the feature mapping results to generate preliminary animation parameters.

[0102] After generating the self-attention mapping results, the system further executes the multi-head attention mechanism to ensure that the multi-dimensional information in the feature map can be fully understood from different angles.

[0103] The multi-head attention mechanism is an extension of the self-attention mechanism. The mapping result is processed by multiple independent attention heads, each of which focuses on different aspects of the fused features from different angles. The feature vectors generated by each head after independent processing will be combined to form the final attention mapping result.

[0104] Specifically, multi-head processing is first performed. The mapped features are divided into several subspaces, and the self-attention mechanism is applied to each subspace. Each attention head performs independent attention calculations in different feature subspaces. Then, the results are combined. The attention outputs of each head are spliced ​​together, and the final feature map is obtained through linear transformation. Multi-head attention processing can capture the relationship between different features from multiple angles, improving the model's ability to capture details.

[0105] After that, the preliminary animation parameters are generated. After the multi-head attention mechanism is processed, the system generates a high-dimensional feature representation that is highly correlated with speech features and emotional features. Based on these feature representations, the system converts them into preliminary animation parameters through a fully connected layer, including key information for driving the expression and lip shape changes of the virtual object, such as the amplitude of facial muscle movement and the shape of lip shape changes.

[0106] Finally, a multi-frame time series analysis is performed on the preliminary animation parameters to obtain the smoothed controller parameters. After generating the preliminary animation parameters, the system needs to ensure that the expression and mouth shape of the virtual object change continuously and naturally. Therefore, the system introduces multi-frame time series analysis to process the preliminary animation parameters to eliminate abrupt changes and keep the movement of expression and mouth shape smooth in the time dimension. Multi-frame time series analysis considers the changing trend of animation parameters between multiple consecutive frames and uses filtering or interpolation algorithms to smooth these parameters. Specifically, preliminary animation parameters are collected from multiple consecutive animation frames, a time series is constructed, and the changes of each parameter in consecutive frames are analyzed. A low-pass filter (such as Kalman filter or Gaussian filter) is used to smooth the time series to remove the drastic fluctuations caused by single-frame calculation errors or noise. For the unnatural transition between frames, an interpolation method (such as linear interpolation or spline interpolation) is used to fill the inter-frame data to ensure the continuity of the transition. Through time series analysis and smoothing, smoothed controller parameters are generated, which are used to drive the expression and mouth shape of the virtual object in real time to ensure that it is synchronized with the voice and emotional characteristics and is natural and smooth.

[0107] The expression changes of the virtual objects rendered in the embodiments of the present application not only accurately reflect the voice content and emotional state, but also maintain consistency in the time dimension, avoiding incoherent jump-like changes.

[0108] An embodiment of the present application also provides a method for driving a virtual object, using the controller parameters obtained by the method of any of the above embodiments to drive the facial expression and lip shape of the virtual object.

[0109] The present application also provides a controller parameter generation device based on artificial intelligence, such as Figure 7 As shown, it includes: an extraction module 72, which is configured to obtain voice data and emotion data corresponding to the voice data, extract voice features from the voice data, and extract emotion features from the emotion data; a fusion module 74, which is configured to fuse the voice features and the emotion features based on a factor graph to obtain fusion features; a data processing module 76, which is configured to perform animation parameter mapping processing on the fusion features to obtain controller parameters for driving facial expressions and lip shapes of virtual objects.

[0110] It should be noted that: the controller parameter generation device based on artificial intelligence provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the controller parameter generation device based on artificial intelligence provided in the above embodiment and the controller parameter generation method embodiment based on artificial intelligence belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0111] Figure 8 FIG. 1 shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present disclosure. It should be noted that: Figure 8 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0112] like Figure 8 As shown, the electronic device includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage part 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for system operation are also stored. The CPU 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0113] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed.

[0114] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A controller parameter generation method based on artificial intelligence, characterized in that: include: Acquire speech data and emotion data corresponding to the speech data, extract speech features from the speech data, and extract emotion features from the emotion data; fusing the speech feature and the emotion feature based on the factor graph to obtain a fused feature; Animation parameter mapping is performed on the fusion features to obtain controller parameters for driving facial expressions and lip shapes of the virtual object.

2. The method according to claim 1, characterized in that Extract speech features from speech data, including: Performing preliminary signal processing on the voice data, and extracting original voice frames from the processed voice data; The original speech frame is processed by a convolutional neural network to obtain the speech feature.

3. The method according to claim 1, characterized in that Extracting emotional features from the emotional data includes: Performing semantic analysis on the emotion data to generate an emotion intensity matrix; The emotion intensity matrix is ​​input into an emotion recognition network, corresponding emotion features are extracted, and the extracted emotion features are normalized to generate standardized emotion features.

4. The method according to claim 1, characterized in that: The speech feature and the emotion feature are fused based on the factor graph to obtain a fused feature, including: Perform joint probability inference on the speech feature and the emotion feature using the factor graph to generate a fusion feature matrix; Performing dimension reduction processing on the fusion feature matrix to obtain the fusion feature after dimension reduction.

5. The method according to claim 1, characterized in that Animation parameter mapping is performed on the fusion features to obtain controller parameters for driving facial expressions and mouth shapes of the virtual object, including: Inputting the fused features into a feedforward neural network, and the feedforward neural network performs feature mapping on the fused features based on a self-attention mechanism; Perform multi-head attention processing on the feature mapping results to generate preliminary animation parameters; A multi-frame time series analysis is performed on the preliminary animation parameters to obtain the smoothed controller parameters.

6. A method for driving a virtual object, characterized in that: The controller parameters obtained by the method according to any one of claims 1 to 5 are used to drive the facial expression and lip shape of the virtual object.

7. A controller parameter generation device based on artificial intelligence, characterized in that: include: An extraction module is configured to obtain voice data and emotion data corresponding to the voice data, extract voice features from the voice data, and extract emotion features from the emotion data; A fusion module, configured to fuse the speech feature and the emotion feature based on a factor graph to obtain a fusion feature; The data processing module is configured to perform animation parameter mapping processing on the fusion features to obtain controller parameters for driving the facial expressions and mouth shapes of the virtual object.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 6.

9. A computer device, characterized in that: include: Memory and processor, The memory stores a computer program; The processor is used to execute the computer program stored in the memory, and when the computer program is run, the processor is enabled to execute the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images