Driving method and related device

Abstract

According to the driving method and the related device provided by the invention, the driving method is realized based on the target prediction model giving the full convolution architecture provided by the invention, the driving method can be applied to mobile terminals with relatively small performance computing power, such as mobile phones and tablet computers, and the application scene of a voice driving model can be effectively expanded; besides, the target prediction model provided by the invention can change the facial expression of the digital person according to the voice issued by the user and synchronously drive the digital person to make a corresponding trunk posture, so that the sense trueness brought to the user by the digital person is effectively improved.

Description

Technical Field

[0001] The present application relates to the field of voice driving, and in particular to a driving method and related devices. Background Art

[0002] With the advancement of rendering and artificial intelligence technologies, digital avatars are rapidly proliferating in fields such as augmented reality, gaming, and animation. Digital humans are simulated human entities created using computer technology, artificial intelligence, and virtual reality. They possess human-like appearance, intelligence, and emotions, and can communicate and interact with humans through natural language.

[0003] When customizing digital humans, using audio to drive them is a very common scenario. With the current development of neural network technology, users of mobile devices such as smartphones and tablets will eventually demand customized digital humans. However, for devices like phones and tablets that use the ARM architecture, existing voice-driven digital human models are difficult to deploy and lack a sense of realism in the driving effects.

[0004] Therefore, it is necessary to study driving methods with wider adaptability and more realistic character driving effects. Summary of the Invention

[0005] The purpose of this application is to provide a driving method and related devices. The driving method is implemented based on the target prediction model given by the present application with a fully convolutional architecture. It can be applied to mobile terminals with low performance and computing power, such as mobile phones and tablets, and can effectively expand the application scenarios of voice-driven models. In addition, since the target prediction model provided by the present application can change the facial expression of the digital human according to the voice sent by the user, and synchronously drive the digital human to make corresponding torso postures, it effectively improves the sensory realism of the digital human to the user.

[0006] The above-mentioned and other objects are achieved by the features of the independent claims. Further implementations are given in the dependent claims, the description and the drawings.

[0007] In the first aspect, the present application provides a driving method, which includes: preprocessing the first voice information input by the user to obtain a first spectrogram corresponding to the first voice information; processing the first spectrogram using a target prediction model to obtain target expression parameters and target torso posture parameters corresponding to the first voice information, wherein the target prediction model is a fully convolutional neural network model, and the target prediction model is a model trained using target data, and the target data includes multiple data representing the correspondence between voice and facial expressions and / or human body postures; based on the target expression parameters and the target torso posture parameters, driving the digital human to speak and make corresponding expressions and postures.

[0008] In this method, the first voice information input by the user can be a sound sent by the user to the electronic device in real time, or it can be a recording file pre-entered by the user into the electronic device. The target prediction model can predict the expressions and postures that a real person may make when speaking the corresponding voice through their mouth based on the voice data input by the user. The target prediction model can output target expression parameters and target torso posture parameters, where the target expression parameters can include multiple key points of the human face, and the target torso posture parameters can include skeletal key points corresponding to the human body (which can be the upper body or the whole body). The electronic device plays the first voice information input by the user (the electronic device can adjust the timbre of the sound in the original first voice information, or add background music, etc.), and adjusts the facial expression and posture of the digital human based on the expression parameters and torso posture parameters corresponding to the first voice information output by the target prediction model, so that the user feels that the digital human is speaking and can flexibly switch expressions and postures according to the content of the speech.

[0009] It is understandable that the target prediction model is implemented, and the model can be applied to mobile terminals with low performance and computing power, such as mobile phones and tablets, which can effectively expand the application scenarios of the voice-driven model; in addition, since the target prediction model provided by this application can change the facial expressions of the digital human according to the voice sent by the user, and synchronously drive the digital human to make corresponding torso postures, it effectively improves the sensory realism of the digital human to the user.

[0010] In this method, the target prediction model can be deployed in the electronic device (i.e., the client) used by the user, and can be used to process the first voice information input by the user to obtain the expression parameters and torso posture parameters corresponding to the first voice information. Specifically, the target prediction model can be pre-trained by other devices provided by the manufacturer of the electronic device before being deployed to the client, and then deployed to the electronic device. For example, the target prediction model can be stored in the mobile phone used by the user, and the target prediction model can be trained by the manufacturer of the mobile phone using other electronic devices (such as super mobile personal computers or servers), and installed or deployed in the mobile phones it sells.

[0011] Optionally, the electronic device may be a device under the advanced reduced instruction set machine ARM architecture, such as a mobile phone, tablet computer, etc., and the electronic device may also be a device under the non-advanced reduced instruction set machine ARM architecture, such as a supercomputer, desktop computer, etc.

[0012] Optionally, the digital human can be a cartoon digital human or a realistic digital human.

[0013] In combination with the first aspect, in a possible implementation, the target prediction model includes an encoder, an expression decoder, a torso decoder, an expression prediction module and a torso posture prediction module, and the encoder, the expression decoder, the torso decoder, the expression prediction module and the torso posture prediction module are all convolutional neural networks. The target prediction model is used to process the first spectrogram to obtain target expression parameters and target torso posture parameters corresponding to the first voice information, including: extracting features from the first spectrogram by the encoder to obtain a first feature; decoding the first feature by the expression decoder and the torso decoder respectively to obtain a second feature representing the correspondence between the first voice information and the facial expression and a third feature representing the correspondence between the first voice information and the human body posture; processing the second feature by the expression prediction module to obtain the target expression parameter, and processing the third feature by the torso posture prediction module to obtain the target torso posture parameter.

[0014] In this embodiment, the encoder, the expression decoder and the torso decoder are all neural network models.

[0015] The first spectrogram is input to the feature extraction of the encoder. Optionally, the encoder can be a TC-ResNet14-1.5 network model, or other network models with more or fewer stages, such as a Jasper network model, a QuartzNet network model, etc.

[0016] It can be understood that the main function of the encoder is to compress the input data, that is, the first spectrogram, into a low-dimensional representation. This process usually involves feature extraction and dimensionality reduction. During the encoding process, the encoder extracts the most representative information by learning the intrinsic structure and association of the first spectrogram to reduce the dimension of the input information while retaining key features. The first feature output by the encoder is a compressed data representation, which contains the essence of the original data and can be used for further analysis or as input to other models, such as the expression decoder and the torso decoder. The decoder is responsible for decompressing the compressed data output by the encoder, that is, the first feature, back to the form of the original data or a similar representation. The decoder will reconstruct the first feature and restore the structure and content of the original data.

[0017] The features output by the encoder (such as the second features) are simultaneously input into the expression decoder and the torso decoder, and are processed by the expression decoder and the torso decoder respectively to obtain the target expression parameters and the target torso posture parameters.

[0018] In combination with the first aspect, in a possible implementation, the encoder includes at least two encoding modules including a first encoding module, the expression decoder includes at least two expression decoding modules including a first expression decoding module, and the torso decoder includes at least two torso decoding modules including a first torso decoder. The first encoding module is used to extract features from the first spectrogram to obtain a first encoding feature. The first encoding module is the first encoding module included in the encoder, the first expression decoding module is the last expression decoding module of the expression decoder, and the first torso decoding module is the last torso decoding module of the torso decoder. The expression decoder and the torso decoder respectively decode the first feature to obtain a second feature representing the correspondence between the first voice information and the facial expression and a third feature representing the correspondence between the first voice information and the human body posture, including: processing the first encoding feature and the first expression decoding feature by the first expression decoding module to obtain the second feature, and processing the first torso encoding feature and the first expression decoding feature by the first torso decoding module to obtain the third feature. The first expression decoding feature is a feature output by the expression decoding module before the first expression decoding module, and the first torso decoding feature is a feature output by the torso decoding module before the first torso decoding module.

[0019] In this embodiment, the encoder, the expression decoder, and the torso decoder all include the same number of modules.

[0020] The features output by each encoding module in the encoder will serve as the input of the next encoding module until the last encoding module. Taking the encoder as an example, which contains three encoding modules (the first encoding module is the first encoding module, the second encoding module is the second encoding module, and the third encoding module is the third encoding module), the features output by the first encoding module (i.e., the first features) will serve as the input of the second encoding module, and the second encoding module will further extract the features to obtain higher-level semantic features; similarly, the features output by the second encoding module will serve as the input of the third encoding module, and the third encoding module will further extract the features to obtain higher-level semantic features. Multi-layer encoding can capture more abstract and complex features in the spectrogram, thereby improving the model's understanding and processing capabilities of the input data. In addition, multi-layer encoding allows the network to automatically adapt to the characteristics of the input data through learning.

[0021] Similarly, the features output by each decoding module in the expression decoder and the torso decoder will serve as the input of the next decoding module, until the last decoding module outputs the final feature. Taking the expression decoder containing three decoding modules as an example (the third expression decoding module is the first decoding module, the second expression decoding module is the second decoding module, and the third expression decoding module is the first decoding module), the features output by the third expression decoding module will serve as the input of the second expression decoding module, and the second expression decoding module will further extract the features to obtain higher-level semantic features; similarly, the features output by the second expression decoding module will serve as the input of the first expression decoding module, and the first expression encoding module will further extract the features to obtain higher-level semantic features, namely the second features.

[0022] In addition, in this embodiment, the input of each decoding module except the first decoding module in the same decoder includes two parts, namely the features output by the previous decoding module, and the features output by the corresponding encoding module in the aforementioned encoder (that is, the features output by the first encoding module (i.e., the first encoding module) will be output to the second encoding module and will also be input to the last expression decoding module (i.e., the first expression decoding module)). The expression decoding module will connect these two features and further process the connected features to output the new features. It is understandable that the features output by the last expression decoding module (i.e., the first expression decoding module) will be input to the expression prediction module for processing by the expression prediction module to obtain the target expression parameters.

[0023] Similarly, each module in the torso decoder will also process the first input feature in the same way. The feature output by the last torso decoding module (i.e., the first torso decoding module) will be input into the torso posture prediction module for processing by the torso posture prediction module to obtain the target torso posture parameters.

[0024] In this embodiment, by using the features output by each layer of the encoding module as the input of the corresponding decoding module, additional information input can be provided to each decoding module, so that the decoding module can restore the structure and content of the original data to a greater extent when decoding and reconstructing the features, thereby improving the accuracy of the model prediction.

[0025] In combination with the first aspect, in a possible implementation, the target data includes at least one first data, at least one second data, and at least one third data, the first data is data that includes both an expression label and a torso posture label, the second data is data that includes an expression label but does not include a torso posture label, and the third data is data that includes a torso posture label but does not include an expression label, and the target prediction model is obtained by performing incomplete supervised training based on the target data.

[0026] In this embodiment, the target prediction model provided in this application is trained using an incompletely supervised training method. The training set used contains three types of data: 1. A dataset with both expression labels and torso pose labels, i.e., the first data; 2. A dataset with only expression labels but no torso pose labels, i.e., the second data; and 3. A dataset with only torso pose labels but no expression labels, i.e., the third data.

[0027] Optionally, in the training set used to train the above target prediction model, the ratio of these three types of data can be 1:1:1.

[0028] In this embodiment, the present application can decouple the output heads related to expression prediction from the output heads related to torso posture prediction. During the training process, when the training data is the first data, the output heads related to expression prediction and / or the output heads related to torso posture prediction are trained simultaneously; when the training data is the second data, only the output heads related to expression prediction are trained; and when the training data is the third data, only the output heads related to torso posture prediction are trained. The output heads related to expression prediction and / or the output heads related to torso posture prediction are selectively trained using the labels contained in the training data to address the problem of scarce data on voice-driven facial expressions and torsos.

[0029] In combination with the first aspect, in a possible implementation, the incompletely supervised training based on the target data includes: extracting features from the second spectrogram through the encoder to obtain a fourth feature; decoding the fourth feature through the expression decoder and the torso decoder respectively to obtain a fifth feature representing the correspondence between the second voice information and the facial expression and a sixth feature representing the correspondence between the second voice information and the human body posture; processing the fifth feature through the expression prediction module to obtain a first expression parameter, and processing the sixth feature through the torso posture prediction module to obtain a first torso posture parameter, and the second spectrogram is obtained through the second voice information in the sample data; when the training data is the first data, the expression decoder is updated using the first expression parameter, and the encoder and the torso decoder are updated using the first torso posture parameter; when the training data is the second data, the encoder and the expression decoder are updated using the first expression parameter; when the training data is the third data, the encoder and the torso decoder are updated using the first torso posture parameter.

[0030] Similarly, during the training of the target prediction model, the features output by each encoding module in the encoder will serve as the input to the next encoding module, until the last encoding module. The features output by each decoding module in the expression decoder and the torso decoder will serve as the input to the next decoding module, until the last decoding module outputs the final features.

[0031] The second spectrogram is converted from the second speech information contained in a sample data. The fourth feature is the feature finally output by the encoder after processing the second spectrogram during the training process. The fifth feature and the sixth feature are the features finally output by the expression decoder and the torso decoder after processing the fourth feature respectively.

[0032] During the incompletely supervised training of the target prediction model:

[0033] 1) When the sample data has voice information and corresponding expression tags;

[0034] That is, when the training data is the second data, the model only trains the output head related to expression prediction (i.e., the expression prediction module). However, since the sample data itself does not contain a torso pose label, the first torso pose parameter output by the torso pose prediction module for this sample data is not credible. Therefore, the first torso pose parameter output by the torso pose prediction module will be discarded and will not be used for the subsequent reverse update of the encoder and the torso decoder. However, the first expression parameter will be used for the subsequent reverse update of the encoder and the expression decoder.

[0035] 2) When the sample data has voice information and corresponding torso posture labels;

[0036] That is, when the training data is the second data, the model only trains the output head related to torso posture prediction (i.e., the torso posture prediction module). However, since the sample data itself does not contain an expression label, the first expression parameter output by the expression prediction module for this sample data is not credible. Therefore, the first expression parameter output by the expression prediction module will be discarded and will not be used for the subsequent reverse update of the encoder and the expression decoder. However, the first torso posture parameter will be used for the subsequent reverse update of the encoder and the torso decoder.

[0037] 3) When the sample data simultaneously has voice information, corresponding expression labels and corresponding torso posture labels;

[0038] That is, when the training data is the first data, the model simultaneously trains the output heads related to expression prediction and the output heads related to torso posture prediction (i.e., the expression prediction module and the torso posture prediction module). That is, when the training data is the first data, the model only trains the output heads related to torso posture prediction. Since the sample data contains both expression labels and torso posture labels, for this sample data, the first expression parameters and the first torso posture parameters output by the expression prediction module are both credible, so the first expression parameters will be used for the subsequent reverse update of the encoder and the expression decoder, and the first torso posture parameters will be used for the subsequent reverse update of the encoder and the torso decoder.

[0039] During the testing process, the device used to train the above-mentioned target prediction model can determine the corresponding loss function based on the above-mentioned first expression parameters and / or first torso posture parameters. The loss function is used to correct the weight matrix in the convolution layer of the encoding module, the expression decoder and the torso decoder to obtain the optimal weight matrix, so that the difference between the predicted value and the target value output by the target prediction model becomes smaller and smaller.

[0040] In combination with the first aspect, in a possible embodiment, the target prediction model also includes an auxiliary expression prediction module and an auxiliary torso posture prediction module, and the method also includes: processing the fifth feature through the auxiliary expression prediction module to obtain a seventh feature that characterizes the correspondence between the second voice information and the facial expression, and the auxiliary torso posture prediction module is used to process the sixth feature to obtain an eighth feature that characterizes the correspondence between the second voice information and the human body posture; the processing of the fifth feature through the expression prediction module to obtain the first expression parameter includes: when the sample data is the first data, processing the fifth feature and the eighth feature to obtain the first expression parameter; the processing of the sixth feature through the torso posture prediction module to obtain the first torso posture parameter includes: when the sample data is the first data, processing the fourth feature and the seventh feature to obtain the first torso posture parameter.

[0041] In this embodiment, the auxiliary expression prediction module can convolve the fifth feature output by the expression decoder (the last expression decoding module) (e.g., 3×1 convolution) to obtain the seventh feature, and then convolve the seventh feature again (e.g., 1×1 convolution) to obtain the auxiliary expression parameters. The auxiliary torso posture prediction module can convolve the sixth feature output by the torso decoder (the last torso decoding module) (e.g., 3×1 convolution) to obtain the eighth feature, and then convolve the eighth feature again (e.g., 1×1 convolution) to obtain the auxiliary torso posture parameters. The seventh feature can be used together with the fifth feature as input to the torso posture prediction module, and processed by the torso posture prediction module to obtain the first torso posture parameters. Similarly, the eighth feature can be used together with the sixth feature as input to the expression prediction module, and processed by the expression prediction module to obtain the first expression parameters.

[0042] It is understandable that the target prediction model needs to predict the torso posture and facial expression corresponding to the input speech. Therefore, in this embodiment, during the training of the target prediction model, the features output by the auxiliary facial expression prediction module can be used as the input of the torso posture prediction module, and the features output by the auxiliary torso posture prediction module can be used as the input of facial expression prediction. In other words, the two auxiliary prediction modules can provide additional information for the prediction of the subsequent prediction module, and enable the prediction module to simultaneously learn the relationship between speech information, facial expression, and torso posture, effectively improving the accuracy of the output results of the two prediction modules.

[0043] In combination with the first aspect, in a possible implementation, the method further includes: processing the fifth feature through the auxiliary expression prediction module to obtain auxiliary expression parameters, processing the sixth feature through the auxiliary torso posture prediction module to obtain auxiliary torso posture parameters, and when the sample data is the first data or the second data, using the auxiliary expression posture parameters to update the auxiliary expression prediction module, the expression decoder and the encoder; when the sample data is the first data or the third data, using the auxiliary torso posture parameters to update the auxiliary torso posture prediction module, the torso decoder and the encoder.

[0044] In this embodiment, during the back propagation process during the training of the target prediction module, the first expression parameter and the first torso posture parameter are also used to constrain the features output by the auxiliary expression prediction module and the auxiliary torso posture prediction module, respectively, that is, the first expression parameter is used to reversely update the auxiliary expression prediction module, the expression decoder and the encoder, and the first torso posture parameter is used to reversely update the auxiliary torso posture prediction module, the torso decoder and the encoder, so that the loss functions of the expression prediction module and the torso posture prediction module converge more quickly, thereby improving the training efficiency of the model and effectively improving the accuracy of the model's prediction results.

[0045] During the incompletely supervised training of the target prediction model:

[0046] 1) When the sample data has voice information and corresponding expression tags;

[0047] That is, when the training data is the second data, the model trains the output heads related to expression prediction (i.e., the expression prediction module and the auxiliary expression prediction module). However, since the sample data itself does not contain the torso posture label, the auxiliary torso posture parameters output by the auxiliary torso posture prediction module for this sample data are not credible. Therefore, the auxiliary torso posture parameters output by the auxiliary torso posture prediction module will be discarded, and the first torso posture parameters output by the torso posture prediction module will also be discarded. The first torso posture parameters and the auxiliary torso posture parameters will not be used for the subsequent reverse update of the model.

[0048] At this time, the valid information output in the target prediction model is the first expression parameter and the auxiliary expression parameter, and the first expression parameter and the auxiliary expression parameter will be used for subsequent reverse update of the encoder, the expression decoder and the auxiliary expression prediction module.

[0049] 2) When the sample data has voice information and torso posture labels;

[0050] That is, when the training data is the third data, the model trains the output heads related to the trunk posture prediction (i.e., the trunk posture prediction module and the auxiliary trunk posture prediction module). However, since the sample data itself does not contain an expression label, the auxiliary expression parameters output by the auxiliary expression posture prediction module for this sample data are not credible. Therefore, the auxiliary trunk posture parameters output by the auxiliary expression prediction module will be discarded, and the first expression parameters output by the expression prediction module will also be discarded. The first expression parameters and the auxiliary expression parameters will not be used for the subsequent reverse update of the model.

[0051] At this time, the valid information output in the target prediction model is the first torso posture parameter and the auxiliary torso posture parameter, and the first torso posture parameter and the auxiliary torso posture parameter will be used for the subsequent reverse update of the encoder, the torso decoder and the auxiliary torso posture prediction module.

[0052] 3) When the sample data simultaneously has voice information, corresponding expression labels and corresponding torso posture labels;

[0053] That is, when the sample data is the first data, the model trains the output heads related to expression prediction and the heads related to torso posture prediction (i.e., the expression prediction module and the auxiliary expression prediction module, the torso posture prediction module and the auxiliary torso posture prediction module). Since the sample data contains both expression labels and torso posture labels.

[0054] At this time, the valid information output from the target prediction model is the first expression parameter and the auxiliary expression parameter, and the first torso posture parameter and the auxiliary torso posture parameter. The first expression parameter and the auxiliary expression parameter will be used for the subsequent reverse update of the encoder, the expression decoder, and the auxiliary expression prediction module. The first torso posture parameter and the auxiliary torso posture parameter will be used for the subsequent reverse update of the encoder, the torso decoder, and the auxiliary torso posture prediction module.

[0055] During the testing process, the device used to train the above-mentioned target prediction model can correct the weight matrix in the convolutional layer in each module according to the corresponding loss function to obtain the optimal weight matrix, so that the difference between the predicted value output by the target prediction model and the target value becomes smaller and smaller.

[0056] In a second aspect, the present application provides an electronic device, comprising: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method in the first aspect or any possible implementation of the first aspect.

[0057] In a third aspect, the present application provides a chip system, which is applied to an electronic device, and the chip system includes one or more processors, which are used to call computer instructions to enable the electronic device to execute a method as in the first aspect or any possible implementation of the first aspect.

[0058] In a fourth aspect, the present application provides a computer-readable storage medium comprising instructions, which, when executed on an electronic device, enables the electronic device to execute the method in the first aspect or any possible implementation of the first aspect.

[0059] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when the computer program product is run on an electronic device, enables the electronic device to execute the method in the first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A network architecture diagram provided for an embodiment of the present application;

[0061] Figure 2 A scene diagram of a voice-driven digital human provided in an embodiment of the present application;

[0062] Figure 3 An architecture diagram of a target prediction model provided in an embodiment of the present application;

[0063] Figure 4 An architectural diagram of an audio preprocessing module provided in an embodiment of the present application;

[0064] Figure 5 An architectural diagram of a decoding module provided in an embodiment of the present application;

[0065] Figure 6 An architecture diagram of an auxiliary prediction module provided in an embodiment of the present application;

[0066] Figure 7 An architectural diagram of a prediction module provided in an embodiment of the present application;

[0067] Figure 8 A scene diagram of a voice-driven digital human provided in an embodiment of the present application;

[0068] Figure 9 A flowchart of a driving method provided in an embodiment of the present application;

[0069] Figure 10 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0070] The terms used in the following examples of the present application are for the purpose of describing specific embodiments only and are not intended to limit the present application. As used in the specification and appended claims of the present application, the singular expressions "a," "an," "said," "above," "the," and "this" are intended to include plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in the present application refers to and encompasses any or all possible combinations of one or more of the listed items.

[0071] To facilitate understanding, the following first introduces the relevant terms involved in the embodiments of this application.

[0072] (1) Neural Network

[0073] A neural network can be composed of neural units, which can be represented by x s The output of the operation unit with the intercept 1 as input can be:

[0074]

[0075] Where, s = 1, 2, ... n, n is a natural number greater than 1, W s is x s The weight of the neural unit, b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0076] (2) Deep Neural Networks

[0077] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with multiple hidden layers. Based on the location of different layers in a DNN, the neural network within a DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. Layers are fully connected, meaning that any neuron in layer i is connected to any neuron in layer i+1. For each layer, the relationship between its input and output is:

[0078]

[0079] in, is the input vector, is the output vector, is the bias vector, W is the weight matrix (also called coefficient), and α is the activation function.

[0080] The coefficient W is introduced below. In a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscript corresponds to the output third layer index 2 and the input second layer index 4. That is, the coefficient from the kth neuron in the (L-1)th layer to the jth neuron in the Lth layer is defined as Among them, the input layer has no W parameter.

[0081] In deep neural networks, more hidden layers allow them to better capture complex real-world situations. Models with more parameters have higher complexity and can perform more complex tasks.

[0082] (3) Convolutional Neural Networks

[0083] A convolutional neural network (CNN) is a deep neural network with a convolutional architecture. It consists of a feature extractor consisting of convolutional layers and subsampling layers. The feature extractor can be thought of as a filter, and the convolution process can be thought of as convolving a trainable filter with an input image or feature map.

[0084] A convolutional layer is a layer of neurons in a convolutional neural network that performs convolution processing on the input signal. Within a convolutional layer, a neuron can only connect to a subset of neurons in adjacent layers. A convolutional layer typically contains several feature planes, each composed of a rectangular array of neurons. Neurons within the same feature plane share weights, which are referred to as convolution kernels. Convolution kernels can be initialized as matrices of random size, and during convolutional neural network training, the kernels learn to acquire appropriate weights. Furthermore, shared weights have the direct benefit of reducing the number of connections between layers in the convolutional neural network, thereby reducing the risk of overfitting.

[0085] In the present application, the target model for predicting the facial expressions and torso posture of a digital human based on user-input voice data may include an encoder that encodes features in the voice data, an expression decoder and a torso decoder that decode the encoded features, and a prediction model (module) that further convolves the features output by the decoder to output expression prediction results and torso posture prediction results. The above-mentioned encoder, expression decoder, torso decoder, and prediction model may all be CNNs. In other words, the target model provided in the present application may be a fully convolutional neural network.

[0086] (4) Spectrogram

[0087] In the field of audio and speech signal processing, we need to convert the signal into a corresponding spectrogram and use the data on the spectrogram as the feature of the signal.

[0088] A spectrogram represents three-dimensional information on a two-dimensional surface. The horizontal axis represents time, the vertical axis represents frequency, and the color depth indicates the energy of the speech data. This graphical format not only displays the temporal evolution of the speech signal's spectrum but also contains information related to the characteristics of the music signal, such as the changes in frequency-domain parameters such as formants and energy. Therefore, a spectrogram combines the characteristics of both a time-domain waveform and a spectrogram, providing lossless information about the speech signal. Features in a spectrogram, such as horizontal lines and vertical bars, provide detailed information about the speech signal. Horizontal lines (black stripes parallel to the time axis) represent formants, while vertical bars (narrow black bars perpendicular to the time axis) correspond to fundamental pitch. These features are of great significance for phonetics research and speech processing applications. For example, the presence of horizontal lines is a key indicator of voiced speech, while the distance between vertical bars indicates the fundamental pitch frequency. Through its unique two-dimensional representation, a spectrogram provides comprehensive information about the frequency, energy, and time dimensions of a speech signal, making it an indispensable tool in speech analysis, research, and applications.

[0089] The spectrogram formation process includes pre-emphasis of the voice information signal, frame division and windowing, short-time Fourier transform to obtain a spectrogram of each frame signal, rotation and mapping of the spectrogram, splicing the transformed multiple frame spectra to form a spectrogram, etc. For details, please refer to the relevant description of the subsequent embodiments, which will not be repeated here. In the embodiment of the present application, after the electronic device collects the voice information input by the user, the electronic device will also convert the voice information into a spectrogram according to the above process, and then use the spectrogram as the input of the target prediction model provided by the present application. The target prediction model can extract and predict features from the spectrogram to obtain expression parameters and torso posture parameters corresponding to the voice information input by the user. The expression parameters and torso posture parameters can respectively represent the facial expressions and body postures that a real human may make when speaking the above voice information through the mouth. Accordingly, the electronic device can use the expression parameters and torso posture parameters to drive the digital human to speak with the predicted expression and make corresponding gestures.

[0090] (5) Loss function

[0091] Training a deep neural network is the process of learning the weight matrix. The ultimate goal of training is to obtain the weight matrices of all layers of the trained deep neural network. The weight matrices of all layers are the weight matrices formed by the vector W of each layer.

[0092] In the process of training a deep neural network, because we want the output of the deep neural network to be as close as possible to the value we really want to predict, the electronic device can compare the current network's predicted value with the desired target value, and then adjust the weight vector of each layer of the neural network based on the difference between the two. Of course, there is usually an initialization process before the first adjustment, that is, pre-configuring parameters for each layer in the deep neural network. For example, if the network's predicted value is too high, the electronic device adjusts the weight vector to make it predict lower, and continues to adjust until the deep neural network can predict the desired target value or a value very close to the desired target value.

[0093] Therefore, electronic devices need to predefine how to compare the difference between predicted and target values. This is known as the loss function or objective function. Loss and objective functions are important equations used to measure the difference between predicted and target values. For example, a higher loss value indicates a greater difference. The training process of a deep neural network is essentially about minimizing this loss.

[0094] (6) Backpropagation algorithm

[0095] Deep neural networks can use a backpropagation (BP) algorithm, a loss function, to correct the initial weight matrix during training, minimizing the difference between the predicted and target values. Specifically, forward propagation of the input signal to the output generates an error loss. This error loss information is then backpropagated to adjust the initial weight matrix, ultimately converging the error loss. The BP algorithm uses the error loss as the primary mechanism for backward propagation, aiming to obtain the optimal weight matrix.

[0096] (7) Incompletely supervised training

[0097] Incomplete supervision means that in the training set used for model training, only a small subset of data has labels (i.e., labeled data), and the rest of the data has no labels (i.e., unlabeled data). This situation occurs in various tasks. Since in incomplete supervision training, only a small amount of data in the dataset has labels, these labeled data are actually not enough to train a good model. However, there is a large amount of unlabeled data available in the dataset. The task of the model is to learn from the training data D = {(u1,v1), (u2,v2),… (u s ,v s ),u s+1 ,u s+2 …u r}. Among them, s samples are labeled and (rs) samples are unlabeled.

[0098] In addition, during the training of some multi-task models, a sample in the training set may have multiple labels. When there is incomplete labeling of a part of the sample data in the training set (that is, the sample data only has some of the labels among the above multiple labels or does not have any of the labels among the above multiple labels), the supervised training carried out by the above training set is also incomplete supervised training. For example, the target prediction model provided in this application can be based on the output of the voice information input by the user and the expression parameters and torso posture parameters corresponding to the voice information, then the training set used to train the target prediction model can be D1 = {(u1, v1, w1), (u2, v2, w2), … (u s ,v s ,w s ),(u s+1 ,v s+1 )(u s+2 ,v s+2 )…(u s+r ,v s+r ),(u s+r+1 ,w +1 )(u s++r+2 ,w s+2 )…(us+r+p ,w s+r+p )}, where u represents the feature, v represents the expression label, and w represents the torso posture label. It can be understood that in the training set D1, the sub-training set {(u1,v1,w1),(u2,v2,w2),…(u s ,v s ,w s )} are datasets with complete labels, which include both expression labels and torso posture labels; sub-training set {(u s+1 ,v s+1 )(u s+2 ,v s+2 )…(u s+r ,v s+r )} is a dataset that only has expression labels but no torso posture labels, and the sub-training set {(u s+r+1 ,w +1 )(u s++r+2 ,w s+2 )…(u s+r+p ,w s+r+p )} is a dataset that only has torso posture labels but no expression labels.

[0099] In the application, the target prediction model can be obtained by performing incompletely supervised training using a deep learning algorithm used in training data that is a mixture of labeled data and unlabeled data.

[0100] With the development of rendering and artificial intelligence technology, digital images have mushroomed in the fields of AR, games, animation, etc. Digital humans refer to human-like simulated entities created using computer technology, artificial intelligence, virtual reality and other technologies. They can have human-like appearance, intelligence and emotions, and can communicate and interact with humans through natural language interaction. The current mainstream neural network models that use voice to drive digital humans focus on predicting the expressions of digital humans, and most of them are based on the Transformer structure (Transformer is a network model that does not include traditional CNN and RNN. Its entire network structure is completely composed of attention mechanisms). For details, please refer to Figure 1 .

[0101] Figure 1 This is a network architecture diagram of a network model for driving digital humans to speak. Figure 1As shown, the network model can include a temporal convolutional network (TCN), a linear interpolation module, a self-attention module, a feed forward neural network, a linear projection module and a Transformer model.

[0102] First, the number of temporal convolutional neural networks can be T. After collecting speech information, the speech information can be divided into T temporally continuous speech segments and respectively input into the T temporal convolutional neural networks. The T temporal convolutional neural networks respectively extract features from the T speech segments, and the output features are simultaneously transmitted to the linear interpolation module, the multi-head self-attention module, the feedforward neural network, and the linear projection module for processing to obtain temporally continuous speech features corresponding to the T speech segments. Where T is an integer greater than 1, and T represents the number of speech segments in the current speech block.

[0103] As described above, the network model performs feature extraction on T sequentially connected speech segments. The extracted speech features for each segment include both the characteristics of the segment itself and the inherent correlations between it and its adjacent segments. Next, the Transformer model within the network model processes the speech features of each segment in the current speech block sequentially to obtain the facial deformation parameters for the current speech block. Facial deformation parameters can include the opening and closing parameters and shape parameters of characteristic features such as the mouth, eyes, and eyebrows.

[0104] Specifically, the Transformer model can process the speech features of the first speech segment in any speech block and the speech features of each speech segment that precedes the first speech segment in time to obtain the facial morphing parameters of the first speech segment. Here, the first speech segment can be any speech segment in the current speech block. If the first speech segment is the speech segment that precedes the first speech segment in time in the current speech block, the speech features of the first speech segment can be directly processed to obtain the facial morphing parameters of the first speech segment.

[0105] Specifically, the Transformer model can include multiple layers connected in sequence, and the facial deformation parameters of each speech segment in the current speech block are obtained in the following manner: the processing of the current layer in the Transformer model is performed on the previous latent features of the first speech segment and the previous latent features of each speech segment that precedes the first speech segment in time sequence to obtain the current latent features of the current speech segment; in this way, after sequentially executing the processing of each layer in the Transformer model, the facial deformation parameters of T speech segments can be obtained. Among them, when the current layer is the first layer of the Transformer model, the previous latent features are speech features; when the current layer is not the first layer of the Transformer model, the previous latent features are obtained by the layer before the current layer; when the current layer is the last layer of the Transformer model, the second latent features are the facial deformation parameters corresponding to the current speech segment.

[0106] Finally, the electronic device deployed with the above network model can drive the digital human to speak based on the above facial deformation parameters and the above voice information. When the digital human outputs the corresponding voice segment through its mouth, the expression parameters corresponding to the voice segment are combined to control the digital human to make corresponding expressions.

[0107] Figure 2 (A) shows the electronic device using Figure 1 The network model shown in the speech-driven digital human speaking scenario.

[0108] like Figure 2 As shown in (A), the electronic device 20 can be a desktop computer, a laptop computer, or other device that is not based on an advanced RISC machine (ARM) architecture. Such devices have good compatibility with the Transformer model. Figure 1 The network model shown in FIG, and the current electronic device has created a digital human 201 according to the user's instructions and displayed the digital human 201 on the screen. Figure 2 As can be seen from (A) in the figure, the digital human 201 created and displayed by the electronic device 20 only includes the human body structure including the head, neck and part of the chest, and does not include the human body structure such as hands and abdomen for presenting specific human postures.

[0109] Before the user sends a voice message to the electronic device 20 to drive the digital human 201 to speak, the digital human 201's facial expression is calm. However, before the user sends the voice message to the electronic device 20 to drive the digital human 201 to speak, the digital human 201's facial expression begins to become joyful. This is because the network model learns from the voice message sent by the user that a real human is likely to express joy and happiness when speaking the voice message. Therefore, the digital human will produce a joyful expression based on the expression parameters output by the network model after processing the voice message. Specifically, the expression parameters can be multiple key points corresponding to facial expressions. These key points can locate specific parts of the face, such as eyebrows, eyes, nose, mouth, and facial contours, to determine the facial expression. Afterwards, the electronic device 20 can adjust the expression of the digital human 201 according to the expression parameters output by the network model, so that the digital human 201 can make responsive expressions while speaking (the digital human speaking can be compared to dubbing an anime character. In fact, the electronic device 20 outputs the words corresponding to the above-mentioned voice information through its microphone. For example, when the voice information sent by the above-mentioned user is the user saying "I am so happy today", the electronic device 20 will also output the sound of "I am so happy today" through the microphone and control the digital human 201 to make corresponding expressions, making it look like the digital human is speaking).

[0110] For some devices with higher performance computing power, such as Figure 2 As for devices such as desktop computers, laptops, and servers under non-ARM architecture (such as Intel x86 architecture) shown in (B), these devices have good compatibility with Transformer-based models and are relatively easy to deploy. Moreover, because these devices have excellent performance, they can be successfully deployed on these electronic devices. Figure 1 The network model shown in the figure is used to drive the digital human created based on the network model to speak and make corresponding expressions. Figure 2 As shown in (B) in the figure, most of the most common smart devices are based on the ARM architecture, including mobile phones, tablets, laptops, and car terminals. For these electronic devices based on the ARM architecture, users may also need to create and support voice-driven digital humans through the devices during use. However, the performance of electronic devices based on the ARM architecture is relatively weak, and the computing power is limited. Figure 1 The network model in is based on the Transformer structure, which has a high time complexity. Figure 1The network model shown is also difficult to deploy on electronic devices based on the ARM architecture. Even after successful deployment, the electronic devices may not be able to smoothly drive the digital human they create based on the user's voice due to performance limitations.

[0111] also, Figure 1 The network model shown only considers the expressions of the digital human when speaking, e.g. Figure 2 In the digital human 201 shown in (A), the electronic device 20 only creates the head, neck, and part of the chest when creating the digital human. When voice-driven digital human 201 speaks, digital human 201 can only change its facial expressions according to the voice sent by the user. However, in reality, the demeanor of a normal human speaking should be reflected in both the facial expression and the posture of the human body (such as hand gestures, forward leaning or swinging of the body, etc.). In particular, in some specific digital human application scenarios, such as voice-driven digital humans giving speeches or driving digital humans to communicate face-to-face with children, the posture of the digital human body often conveys emotions better than facial expressions and attracts the user's attention. Therefore, for such scenarios, the model used to drive the digital human speech with voice should be able to drive the movement of the face and torso simultaneously, driving the digital human to make corresponding expressions and gestures. However, in actual algorithm design, data where voice drives facial expressions and torso movements simultaneously is very rare, making it difficult to train a network model that can drive the movement of the face and torso simultaneously.

[0112] Therefore, how to successfully apply the network model that drives the digital human to mobile terminals with lower performance, and enable the network model to drive the synchronous movement of the digital human's face and torso based on the voice sent by the user to achieve a more realistic driving effect, is a difficult problem in current technological development.

[0113] To address the shortcomings of the aforementioned network model, this application provides a driving method and related apparatus. To implement this method, a target prediction model provided by this application can be installed on devices with low computing power, such as mobile phones and tablets. This target prediction model can be a fully convolutional neural network model obtained through incompletely supervised training. After a user sends a voice message to an electronic device, the electronic device can process the voice message using the target prediction model to obtain expression parameters and torso posture parameters corresponding to the voice message. The electronic device can then drive the digital human to speak and make corresponding expressions and gestures based on these expression parameters and torso posture parameters.

[0114] First, the target prediction model provided by this application is introduced. It should be noted in advance that the target prediction model provided by this application can be deployed in the electronic device (i.e., the client) used by the user, and it can be used to process the voice information input by the user to obtain the expression parameters and torso posture parameters corresponding to the voice information. Specifically, the above-mentioned target prediction model may be pre-trained by other devices provided by the manufacturer of the electronic device before being deployed to the client, and then deployed to the electronic device. For example, the above-mentioned target prediction model may be stored in the mobile phone used by the user, and the target prediction model may be trained uniformly by the manufacturer of the above-mentioned mobile phone using other electronic devices (such as super mobile personal computers or servers), and installed or deployed to the mobile phones it sells.

[0115] Please refer to Figure 3 , Figure 3 This is the architecture diagram of the target prediction model provided by this application. Figure 3 As shown, the target prediction model can include an audio preprocessing module 301, an encoding module x1, an encoding module x2, an encoding module x3, an expression decoding module y1, an expression decoding module y2, an expression decoding module y3, a torso decoding module z1, a torso decoding module z2, a torso decoding module z3, an auxiliary expression prediction module 302, an auxiliary torso posture prediction module 303, an expression prediction module 304 and a torso posture prediction module 305.

[0116] Next, combine Figure 3 This paper introduces the training process of the target prediction model provided by this application.

[0117] First, in the training process of the above-mentioned target prediction model, the input data voc1 can be a training set obtained in advance. The training set contains multiple sample data, any of which contains a voice message, and the expression and / or torso posture that a real human may make when speaking the voice message (expression and torso posture can exist in the form of parameters, and these two parameters are the labels corresponding to the sample data). Since the training method adopted by the target prediction model provided in this application is incompletely supervised training, the sample data contained in the above-mentioned training set during the training process can be divided into three categories: 1) data with only expression labels; 2) data with only torso posture labels; 3) data with both expression and torso posture labels.

[0118] For any piece of sample data, the audio preprocessing module 301 preprocesses the speech information contained in the sample data to obtain a spectrogram spe1 corresponding to the speech information.

[0119] Figure 4The specific steps involved in the audio preprocessing module preprocessing the speech information contained in the sample data are shown as an example. Figure 4 As shown, the pre-processing operation may include: a format conversion module 401 , a voice activity module 402 and a voice noise reduction module 403 .

[0120] Specifically, when the voice information voc2 contained in any sample data is input into the network model, the format conversion module 401 converts the original format (such as .mp3 or .aiff) into a digital signal in .wav format for processing. Optionally, the format conversion module 401 is also responsible for sampling and quantizing the voice information voc2, that is, converting the continuous-time voice signal into a discrete-time voice signal, and obtaining discrete samples by measuring the voice signal at a series of time points. In the process of converting the continuous amplitude signal into a discrete amplitude signal, the amplitude of the continuous signal is divided into a finite number of discrete levels, and each sample is mapped to the closest discrete level.

[0121] The voice activity module 402 then processes the voice information converted by the format conversion module using a voice activity detection (VAD) algorithm to distinguish speech signals from non-speech signals (such as silence and pure noise) contained in the voice information. Specifically, the voice activity module 402 performs pre-emphasis and frame windowing on the voice information, calculates the short-term energy and zero-crossing rate of each frame, performs a moving average filter on the short-term energy and zero-crossing rate, performs threshold detection to distinguish speech frames from non-speech frames, performs connectivity analysis, and performs smoothing.

[0122] It is understandable that because the power spectrum of the speech signal decreases with increasing frequency, most of the energy of the speech is concentrated in the low-frequency part, resulting in a very low signal-to-noise ratio in the high-frequency part. Therefore, it is necessary to pre-emphasize the speech information. Specifically, a first-order high-pass filter can be used to improve the signal-to-noise ratio of the signal in the high-frequency part. Through the frame division and windowing operation, the speech information can be converted from a one-dimensional signal to a two-dimensional signal; the above-mentioned speech signal can be framed in 10ms to 30ms. In order to ensure smooth transition and continuity between frames, there can be overlap between frames. The short-time energy calculation can perform a square operation on the speech signal of each frame to obtain short-time energy. This is because the energy of the speech signal is proportional to the square of its amplitude. The square operation can highlight the amplitude information of the signal, thereby better extracting the characteristics of the speech signal. Specifically, the calculation formula for short-time energy is:

[0123]

[0124] Among them, x n(m) represents the amplitude of the n-th frame speech signal at the m-th sampling point, and N is the length of the frame.

[0125] The zero-crossing rate calculation is to count the number of times each frame of speech signal transitions from positive to negative or from negative to positive, that is, to calculate the short-term zero-crossing rate. The formula for calculating the zero-crossing rate is:

[0126]

[0127] Among them, sgn[x n (m)] represents the sign function value of the nth frame at the mth sampling point, which is used to determine whether the signal crosses the zero point.

[0128] Moving average filtering is to perform moving average filtering on the calculated short-time energy and zero-crossing rate to reduce the impact of random noise and make the signal smoother. Specifically, moving average filtering can be achieved by averaging multiple consecutive frames of data. Threshold detection and differentiation, that is, according to the statistical characteristics of short-time energy and zero-crossing rate, two thresholds are set, a high threshold and a low threshold. By comparing the short-time energy and zero-crossing rate of the current frame with the set threshold values, it can be determined whether the frame belongs to a speech signal or noise. In this way, speech signals can be effectively distinguished from background noise.

[0129] Next, the speech noise reduction module 403 can reduce the noise of the speech signal output by the speech activity module 402 through the noise reduction algorithm to obtain the spectrogram to be input into the network. Where t is time, f is frequency, and t and f are obtained based on the frame windowing settings in the VAD algorithm and the sampling rate during format conversion.

[0130] Optionally, in the embodiment of the present application, t=500ms, f=80Hz. Optionally, the noise reduction algorithm may be any one of spectral subtraction, Wiener filtering, adaptive filtering, mean filtering, and median filtering.

[0131] It can be understood that in the embodiments of the present application, the window size of the frame windowing selected in the VAD algorithm and the threshold value during threshold detection can be set to other values, and the noise reduction algorithm used for noise reduction can also be other noise reduction algorithms. This application does not limit this.

[0132] Next, the above spectrogram It will be further input into the backbone network serving as an encoder, namely encoding module x1, encoding module x2, and encoding module x3 for feature extraction. Optionally, the backbone network may be a TC-ResNet14-1.5 network model. The TC-ResNet14-1.5 network model is a fast and accurate convolutional neural network comprising three stages. Optionally, the backbone network may also be other backbone networks comprising more or fewer stages, such as a Jasper network model, a QuartzNet network model, etc., which is not limited in this application.

[0133] Next, we will take the above backbone network as an example, which is a TC-ResNet14-1.5 network model consisting of three stages. Figure 3 As shown, the above spectrogram First, it will be input into the first encoding module contained in the above backbone network, that is, encoding module x1. Next, the features output by each encoding module will serve as the input of the next encoding module, until encoding module x3 outputs the corresponding features. In other words, the features output by encoding module x1 will serve as the input of encoding module x2, and encoding module x2 will further extract the features to obtain higher-level semantic features; similarly, the features output by encoding module x2 will serve as the input of encoding module x3, and encoding module x3 will further extract the features to obtain higher-level semantic features. It can be understood that the above spectrogram After passing through the three encoding modules of the backbone network, the intermediate features of the three different stages are generated. For the i-th stage (i∈{1,2,3}), its characteristics are Among them, c i Represents the encoding module x i (i∈{1,2,3}) represents the output channel for the output feature. Specifically, for the TC-ResNet14-1.5 model, [c1,c2,c3] = [24,36,48]. Multi-layer encoding can capture more abstract and complex features in the spectrogram, thereby improving the model's understanding and processing capabilities of the input data. Furthermore, multi-layer encoding allows the network to automatically adapt to the characteristics of the input data through learning.

[0134] Next, f iThe input is input into the corresponding decoding module. The target prediction model provided in this application includes two decoders, namely the expression decoder and the torso decoder, which are respectively used to decode the features used to predict expression / torso posture. Among them, the expression decoder is expression decoding module y1, expression decoding module y2 and expression decoding module y3, and the torso decoder corresponds to torso decoding module z1, torso decoding module z2 and torso decoding module z3. In the same decoder, the input of each decoding module except the first decoding module (such as expression decoding module y3 and torso decoding module z3) includes two parts, namely the features output by the corresponding encoding module in the above-mentioned backbone network and the features output by the previous decoding module.

[0135] Assume that the expression decoding module y j The input feature of (j∈{1,2,3}) is i j , whose output feature is o j , then for the expression decoding module y3, its input feature i3 is f3, and f3 is the feature output by the encoding module a3. For other expression decoding modules, their input is:

[0136] i j =cat(o j+1 ,f j );

[0137] Where cat(.) is the channel connection function. Assuming that the size of the input feature of each module is a, the input and output tensor changes are:

[0138]

[0139] cin j Represents the expression decoding module y j Input channel, cout j Represents the expression decoding module y j In the embodiment of the present application, the output channels of the expression decoding module y3, the expression decoding module y2, and the expression decoding module y1 are c2, c3, and 2c3 respectively.

[0140] Similarly, assuming that the trunk decoding module z n The input feature of (n∈{1,2,3}) is m n , whose output feature is p n , then for the trunk decoding module z3, its input feature m3 is f3, and f3 is the feature output by the encoding module a3. For other trunk decoding modules, their input is:

[0141] m n =cat(p n+1 ,f n );

[0142] Where cat(.) is the channel connection function. Assuming that the size of the input feature of each module is b, the input and output tensor changes are:

[0143]

[0144] cin n Represents the trunk decoding module z n Input channel, cout n Represents the trunk decoding module z j In the embodiment of the present application, the output channels of the trunk decoding module z3, the trunk decoding module z2 and the trunk decoding module z1 are c2, c3 and 2c3.

[0145] Optionally, for any decoding module contained in the expression decoder and the above-mentioned trunk decoder (i.e., any one of the expression decoding module y1, expression decoding module y2, expression decoding module y3, trunk decoding module z1, trunk decoding module z2, and trunk decoding module z3), its detailed structure can refer to Figure 5 .

[0146] like Figure 5 As shown, decoding module 50 can be any one of expression decoding module y1, expression decoding module y2, expression decoding module y3, trunk decoding module z1, trunk decoding module z2, and trunk decoding module z3; feature 51 can be any one of features i1, i2, i3, m1, m2, and m3 input by expression decoding module y2, expression decoding module y3, trunk decoding module z1, trunk decoding module z2, and trunk decoding module z3; feature 52 can be any one of features o1, o2, o3, p1, p2, and mp3 output by expression decoding module y2, expression decoding module y3, trunk decoding module z1, trunk decoding module z2, and trunk decoding module z3. For example, when decoding module 50 is expression decoding module y2, feature 51 can be input feature i2 of expression decoding module y2, i.e., cat(o3, f2), and feature 52 can be output feature o3 of expression decoding module y2, and so on. No further details will be given here. The decoding module 50 may include an upsampling module 501, a convolution module 502, and an activation module 503, wherein:

[0147] The upsampling module 501 can amplify the feature 51. Optionally, the amplification factor provided by the upsampling module 501 can be 2, that is, the width and height of the feature are both increased to twice the original. It is understood that the size of the feature may need to be adjusted to adapt to different input or output requirements. Through bilinear upsampling, the spatial dimension of the feature map can be increased while maintaining the semantic information of the feature map, which is beneficial to improving the performance and accuracy of the model.

[0148] The convolution module 502 can perform a convolution operation on the features amplified by the upsampling module 501 to obtain more critical features. Optionally, the convolution module 502 can be a one-dimensional convolution layer with a size of 3×1.

[0149] The activation module 503 is used to convert the features output by the convolution module 502 into new output values. It is understood that the features output by the convolution module 502 are actually linear, and the activation function is used to map the output of the neurons contained in the convolutional neural network (i.e., the convolution module 502) to a new value, which can be used for calculations by the neurons in the next layer (e.g., the next decoding module and the subsequent auxiliary expression prediction module). Optionally, the activation function contained in the activation module 503 can be a ReLU function.

[0150] In some embodiments, the upsampling operation performed by the upsampling module 501 may also be other types of upsampling operations, such as transposed convolution upsampling and nearest neighbor upsampling; in addition, the convolution layer contained in the convolution module 502 may also be a convolution layer of another size, or a deep convolution network obtained by stacking convolution layers of various sizes; in addition, the activation function used by the activation module 503 may also be other activation functions, such as a sigmoid function, a tanh function, etc., which are not limited in this application.

[0151] In addition, it should be noted that in the target prediction model provided in the embodiment of the present application, the network structures of the expression decoder and the torso decoder can be basically the same, the only difference being the final output feature information. Figure 5 When the decoding module 50 in is the expression decoding module y3, the feature 51 finally output by the expression decoder represents the relationship between speech and expression; when Figure 5 When the decoding module 50 in is the trunk decoding module z3, the feature 51 finally output by the trunk decoder represents the relationship between the speech and the trunk posture.

[0152] Afterwards, the features finally output by the above two decoders will be further transmitted to the two auxiliary prediction modules contained in the prediction model, namely the auxiliary expression prediction module 302 and the auxiliary torso posture prediction module 303. These two modules will respectively perform preliminary prediction of expression and / or torso posture information, and provide additional prediction information for the final expression prediction module 304 and torso posture prediction module 305, thereby enhancing the accuracy of the final prediction results.

[0153] Figure 6 The detailed architecture of the auxiliary expression / torso posture prediction module is shown in FIG. Figure 6 As shown, the auxiliary prediction module 60 can be Figure 3In the auxiliary expression prediction module 302 or the auxiliary trunk posture prediction module 303. In the case where the auxiliary prediction module 60 is the auxiliary expression prediction module 302, Figure 6 The feature 61 in is the feature o1 output by the expression decoding module y1; accordingly, when the auxiliary prediction module 60 is the auxiliary torso posture prediction module 303, Figure 6 The feature 61 in is the feature p1 output by the torso posture module z1. Its output channels are all 2c3.

[0154] The auxiliary prediction module 60 may include an upsampling module 601, a convolution module 602, an activation module 603, and a convolution module 604, wherein:

[0155] The upsampling module 601 may amplify the feature 61. Optionally, the amplification factor provided by the upsampling module 601 may be 2, that is, the width and height of the feature are both increased to twice their original values.

[0156] The convolution module 602 can perform a convolution operation on the features amplified by the upsampling module 601 to obtain a more critical feature 62. Optionally, the convolution module 602 can be a one-dimensional convolution layer of size 3×1. In addition to being input into the activation module 603, the features output by the convolution module 602 will be further input into the expression prediction module 304 or the torso posture prediction module 305 to provide additional prediction information for the final prediction process of the expression prediction module 304 or the torso posture prediction module 305, thereby improving the accuracy of the final prediction result. Specifically, in the case where the auxiliary prediction module 60 is the auxiliary expression prediction module 302, the feature 62 will subsequently serve as the input of the torso posture prediction module 305; correspondingly, in the case where the auxiliary prediction module 60 is the auxiliary expression prediction module 302, the feature 62 will subsequently serve as the input of the expression prediction module 304.

[0157] The activation module 603 is used to convert the features output by the convolution module 602 into new output values. Optionally, the activation function included in the activation module 603 can be a ReLU function.

[0158] The convolution module 604 can further perform convolution operations on the features output by the activation module and activate them using an activation function (such as a ReLU function) to obtain output parameters 63. Optionally, the convolution module 604 can be a one-dimensional convolution layer of size 1×1. Specifically, when the auxiliary prediction module 60 is the auxiliary expression prediction module 302, the output parameter 63 can be Figure 3; correspondingly, when the auxiliary prediction module 60 is the auxiliary torso posture prediction module 303, the output parameter 63 can be the auxiliary torso posture parameter pos0. Specifically, during the backpropagation process of the training process, the output parameter 63 is also used to constrain the features output by the convolution module 602, that is, the auxiliary parameters can be used to update the parameter weights in the corresponding convolution layer of the convolution module 602, so as to make the features output by the auxiliary prediction module (i.e., the features subsequently input to the expression prediction module 304 and the torso posture prediction module) more accurate, thereby improving the accuracy of the final prediction result.

[0159] Similarly, it should be noted that in the target prediction model provided in the embodiment of the present application, the network structures of the auxiliary expression prediction module 302 and the auxiliary torso posture prediction module 303 can be basically the same, and the only difference is that the final output feature information is different. It is understandable that due to the different prediction contents, the 1×1 convolutional layer contained in the auxiliary expression prediction module 302 and the auxiliary torso posture prediction module 303 (refer to Figure 6 The output dimension of the convolution module 604 in the image is also different. For the auxiliary expression prediction module 302, since it predicts auxiliary expressions, the output dimension of the auxiliary expression prediction module 302 is related to the selected facial expression parameter model; in the embodiment of the present application, the blendshape parameter model can be selected to control the facial movement of the digital human. Then the output channel of the above 1×1 convolution layer is 52, and the auxiliary expression prediction result (i.e., exp0) is For the auxiliary torso posture prediction module 303, since it predicts the auxiliary torso posture, the output dimension is related to the selected human body parameter model. In the embodiment of the present application, the SMPL-X upper body parameter model can be selected to control the body movement of the digital human. At this time, the output channel of the above 1×1 convolution layer is 78, and the auxiliary torso posture prediction result (i.e., pos0) is

[0160] In some embodiments, the upsampling operation performed by the upsampling module 601 may also be other types of upsampling operations, such as transposed convolution upsampling and nearest neighbor upsampling; in addition, the convolution layer contained in the convolution module 602 may also be a convolution layer of another size, or a deep convolution network obtained by stacking convolution layers of various sizes, and its output channel may also be other channels; in addition, the activation function used by the activation module 603 may also be other activation functions, such as the sigmoid function, the tanh function, etc. In some embodiments, the above-mentioned blendshape parameter model may also be replaced with other parameter models, such as the FLAME model; similarly, the above-mentioned SMPL-X upper body parameter model may also be replaced with other parameter models, or the prediction range may be increased (i.e., replaced with the SMPL-X full body parameter model), and this application does not limit this.

[0161] Afterwards, combined with the above description, it can be seen that one input of the torso posture prediction module 305 is the feature (refer to the aforementioned feature 62) output by the first convolution module (refer to the aforementioned convolution module 602) in the auxiliary expression prediction module 302, and the other input is the feature finally output by the expression decoder, that is, the feature o1 output by the expression encoding module y1. One input of the expression prediction module 304 is the feature (refer to the aforementioned feature 62) output by the first convolution module (refer to the aforementioned convolution module 602) in the auxiliary torso posture prediction module 303; the other input is the feature finally output by the torso encoder, that is, the feature p1 output by the torso decoding module z1. The significance of the expression / torso posture prediction module is to use the above information to make the final prediction and output the expression parameters and torso posture parameters corresponding to the voice information voc2.

[0162] Figure 7 The detailed architecture of the expression / torso posture prediction module is shown in FIG. Figure 7 As shown, the prediction module 70 can be Figure 3 In the expression prediction module 304 or the trunk posture prediction module 305. In the case where the prediction module 70 is the expression prediction module 304, Figure 7 The decoding module 71 in the expression decoding module y1 can be the expression decoder (expression decoding module y1) in the above description, the auxiliary prediction module 72 can be the auxiliary trunk posture prediction module 303 in the above description, the feature 73 can be the feature o1 output by the expression decoding module y1, the feature 74 can be the feature output by the first convolution module in the auxiliary expression prediction module 302, and the prediction parameter 75 can be Figure 3 Similarly, when the prediction module 70 is the trunk posture prediction module 305, Figure 7 The decoding module 71 in the above description can be the trunk decoder (trunk decoding module z1), the auxiliary prediction module 72 can be the auxiliary expression prediction module 302 in the above description, the feature 73 can be the feature p1 output by the expression decoding module y1, the feature 74 can be the feature output by the first convolution module in the auxiliary trunk posture prediction module 303, and the prediction parameter 75 can be Figure 3 The expression parameter pos1 in .

[0163] The prediction module 70 may include a channel connection module 701, an activation module 702, a convolution module 703, and a convolution module 704, wherein:

[0164] The channel connection module 701 includes a channel connection function cat(.) for connecting feature 73 and feature 74.

[0165] The activation module 702 is used to convert the features output by the channel connection module 701 into new output values. Optionally, the activation function included in the activation module 702 can be a ReLU function.

[0166] The convolution module 703 can perform a convolution operation on the features activated by the activation module 702 and convert the obtained features into new output values ​​using an activation function (e.g., a ReLU function). Optionally, the convolution module 703 can be a one-dimensional convolution layer of size 3×1, and its output channels can be 2c3.

[0167] The convolution module 704 is used to perform a convolution operation on the features that have been convolved and activated by the convolution module 703 and further convolution operation on the features output by the activation module to obtain the prediction parameters 75. Optionally, the convolution module 604 can be a one-dimensional convolution layer of size 1×1. Specifically, in the case where the auxiliary prediction module 70 is the expression prediction module 304, the prediction parameter 75 can be Figure 3 ; correspondingly, when prediction module 70 is torso posture prediction module 305, prediction parameter 75 can be torso posture parameter pos1. Specifically, during the backpropagation process of the training process, prediction parameter 75 is also used to constrain the features output by convolution module 703. That is, prediction parameter 75 can be used to update the parameter weights in the corresponding convolution layer of convolution module 703 to make the prediction results of prediction module 70 more accurate.

[0168] Similarly, it should be noted that in the target prediction model provided in the embodiment of the present application, the network structures of the expression prediction module 304 and the torso posture prediction module 305 can be basically the same, and the difference is only in the final output feature information. It is understandable that due to the different prediction contents, the 1×1 convolutional layer contained in the expression prediction module 304 and the torso posture prediction module 305 (refer to Figure 7 The output dimension of the convolution module 704 in the image is also different. For the expression prediction module 304, since it predicts expressions, the output dimension of the expression prediction module 304 is related to the selected facial expression parameter model; in this embodiment of the application, the blendshape parameter model can be selected to control the facial movement of the digital human. The output channel of the above 1×1 convolution layer is 52, and the auxiliary expression prediction result (i.e., exp1) is For the torso posture prediction module 305, since it predicts the torso posture, the output dimension is related to the selected human body parameter model. In the embodiment of the present application, the SMPL-X upper body parameter model can be selected to control the body movement of the digital human. At this time, the output channel of the above 1×1 convolution layer is 78, and the auxiliary torso posture prediction result (i.e., pos1) is

[0169] In some embodiments, the convolutional layers contained in the convolution module 703 may be convolutional layers of other sizes, or a deep convolutional network obtained by stacking convolutional layers of various sizes, and its output channels may also be other channels. In addition, the activation function used by the activation module 702 may also be other activation functions, such as the sigmoid function, the tanh function, etc. In some embodiments, the above-mentioned blendshape parameter model may also be replaced with other parameter models, such as the FLAME model. Similarly, the above-mentioned SMPL-X upper body parameter model may also be replaced with other parameter models, or the prediction range may be increased (i.e., replaced with the SMPL-X full body parameter model). This application does not limit this.

[0170] In addition, combined with the above description, it can be seen that the target prediction model provided in this application is trained using an incompletely supervised training method, and the training set used contains three types of data: 1. A data set that has both expression labels and torso posture labels; 2. Data that only has expression labels but no torso posture labels; 3. A data set that only has torso posture labels but no expression labels. In this application, in the training set used to train the above-mentioned target prediction model, the ratio of these three types of data can be 1:1:1. Specifically, this application can decouple the output heads related to expression prediction (i.e., the expression prediction module 304 and the auxiliary expression prediction module 302) and the output heads related to torso posture prediction (i.e., the torso posture prediction module 305 and the auxiliary torso posture prediction module 303), and adopt the training process accordingly:

[0171] 1) The sample data currently used for training has speech information (spectrogram) and corresponding expression labels

[0172] In this case, the output heads related to expression prediction in model training, namely the auxiliary expression prediction module 302 and the auxiliary torso posture prediction module 303, will still output auxiliary expression parameters and auxiliary torso posture parameters respectively. However, since the sample data does not contain a torso posture label, the auxiliary torso posture parameters output by the auxiliary torso posture prediction module 303 for this sample data are not credible. Therefore, these auxiliary torso posture parameters will be discarded and will not be used for subsequent reverse updates of the model.

[0173] At this time, the relevant information output in the target prediction model is and The loss function at this time is:

[0174]

[0175] Optionally, γ=0.2, λ=0.8.

[0176] 2) The sample data currently used for training has speech information (spectrogram) and corresponding torso posture label

[0177] In this case, the model trains the output head related to torso posture prediction. At this time, the auxiliary expression prediction module 302 and the auxiliary torso posture prediction module 303 still output auxiliary expression parameters and auxiliary torso posture parameters respectively, but the output auxiliary expression parameters are not credible and are therefore discarded and will not be used for subsequent reverse model updates.

[0178] At this time, the relevant information output in the network is and The loss function at this time is:

[0179]

[0180] Optionally, δ=0.2, ρ=0.8.

[0181] 3) The sample data currently used for training contains both speech information (spectrogram) and corresponding expression labels. and the corresponding torso pose labels

[0182] In this case, the model simultaneously trains the output heads related to expression prediction and the output heads related to torso posture prediction. At this time, the auxiliary expression parameters and auxiliary torso posture parameters output by the auxiliary expression prediction module 302 and the auxiliary torso posture prediction module 303 respectively can be used for subsequent reverse updates of the model.

[0183] At this time, the relevant information output in the network is and The loss function at this time is:

[0184]

[0185] Optionally, γ=0.2, λ=0.8, δ=0.2, ρ=0.8.

[0186] In some embodiments, the ratio of the three types of data in the training set can also be other ratio values, and the specific values ​​of the coefficients (i.e., γ, λ, δ, ρ) in the above-mentioned loss functions can also be other values. This application does not limit this.

[0187] Optionally, each of the above loss functions can take L1 and L2 to measure the loss between prediction and label:

[0188] Loss(prediction,label)=αL1(label,prediction)+βL2(label,prediction);

[0189] Optionally, α = 10, β = 100. In some embodiments, α and β may be other values, and the loss function may be other loss functions, which are not limited in this application.

[0190] During the test process, the device used to train the above-mentioned target prediction model can correct the weight matrix in the convolution layer according to the corresponding loss function to obtain the optimal weight matrix, so that the difference between the predicted value and the target value becomes smaller and smaller.

[0191] In addition, in the process of reverse updating of the target prediction model, the embodiment of the present application can select the Adam optimizer to optimize the performance of the model, and the initial learning rate can be set to lr=1e -5 , eps=1e -8 During the model training process, a total of 100 epochs can be trained, and the learning rate decays by 0.1 every 30 epochs. Of course, in some embodiments, the target prediction model provided in this application can also use other optimizers, including but not limited to SGD, RMSprop and other optimizers to optimize the model, and can also use other decay strategies and epoch combinations to train the model, which is not limited in this application.

[0192] In some embodiments, Figure 3 The target prediction model shown can also exclude auxiliary expression prediction module 302 and auxiliary torso pose prediction module 303. In this case, expression prediction module 304 receives only the features output by expression decoding module y1, while torso pose prediction module 305 receives only the features output by torso decoding module z1. Due to the lack of additional information provided by the auxiliary prediction modules, the target prediction model can sacrifice some prediction accuracy to reduce the performance requirements of the target prediction model and achieve faster prediction efficiency.

[0193] Similarly, Figure 3 The application process of the target prediction model shown can refer to Figure 3-Figure 7 The relevant description will not be repeated here. It should be noted that the application of the target prediction model usually occurs after the electronic device is sold to the user. Therefore, during the application process, the input of the target prediction model can be the voice information input by the user (that is, the voice command issued by the user to the electronic device). In addition, since the model can no longer be reversely updated during the application process, Figure 3The auxiliary expression prediction module 302 and the auxiliary torso posture prediction module 303 may not work, or the features output by these two modules will be discarded and no longer serve as inputs to the subsequent expression prediction module 304 and torso posture prediction module 305. At this time, the input to the expression prediction module 304 is only the features output by the expression decoding module y1, and the input to the torso posture prediction module 305 is only the features output by the torso decoding module z1.

[0194] In an embodiment of the present application, the encoding module x1 can be called the "first encoding module", the expression decoding module y1 can be called the "first expression decoding module", the torso decoding module z1 can be called the "first torso decoding module", the encoding modules x1-x3 can be called "at least two encoding modules", the expression decoding modules y1-y3 can be called "at least two expression decoding modules", and the torso decoding modules z1-z3 can be called "at least two torso decoding modules".

[0195] In the application process of the above-mentioned target prediction model, the input of the target prediction model can be called "first speech information", the spectrogram obtained after the first speech information is processed by the audio preprocessing module 301 can be called "first spectrogram", and when the target prediction model processes the first spectrogram, the feature output by the encoding module x3 can be called "first feature", the feature output by the encoding module x1 can be called "first encoding feature", the feature output by the expression decoding module y1 can be called "second feature", the feature output by the expression decoding module y2 can be called "first expression decoding feature", the feature output by the torso decoding module z1 can be called "third feature", the feature output by the torso decoding module z2 can be called "first torso decoding feature", the expression parameters output by the expression prediction module 304 can be called target expression parameters, and the parameters output by the torso posture prediction module 305 can be called "target torso posture parameters".

[0196] In the application process of the above-mentioned target prediction model, the voice information voc2 can be called "second voice information", the spectrogram spe1 can be called "second spectrogram", and the feature 51 can be called "fourth feature". When the decoding module 50 is the expression decoding module y1, the feature 52 can be called "fifth feature". When the decoding module 50 is the torso decoding module z1, the feature 52 can be called "sixth feature", the expression parameter exp1 can be called "first expression parameter", and the torso posture parameter pos1 can be called "first torso posture parameter". When the auxiliary prediction module 60 is the auxiliary expression prediction module 302, the feature 62 can be called "seventh feature". When the auxiliary prediction module 60 is the auxiliary torso posture prediction module 303, the feature 62 can be called "eighth feature".

[0197] Without changing the target prediction model architecture, the data used for training can be changed to supervise the model training process, and the target prediction model provided by this application can be used to perform tasks of its type, such as various voiceprint recognition tasks.

[0198] Figure 8 The figure shows that mobile phones, tablets and other devices have deployed Figure 3 After the target prediction model is shown, the target prediction model is used to voice-drive the digital human to speak.

[0199] like Figure 8 As shown, the electronic device 80 can be a mobile phone, tablet or other device under the ARM architecture, and the electronic device 20 has been successfully deployed Figure 3 The target prediction model shown in , and the current electronic device has created a digital human 801 according to the user's instructions and displayed the digital human 801 on the screen. Figure 8 As can be seen in the figure, the digital human 801 created and displayed by the electronic device 80 includes not only the head and neck, but also the upper body (including the human body structures such as the hands and abdomen for presenting specific human postures).

[0200] like Figure 8 As shown in (A) in the figure, after the user's voice prompts digital human 801 to speak for the first time, its facial expression relaxes (it blinks), and its arms are naturally lowered, with fists clenched in front of its chest. This is because the target prediction model predicts from the user's voice message that a real human is likely to show this expression and make this gesture when speaking the voice message, for example, if the user's voice message contains content such as a funny joke.

[0201] like Figure 8 As shown in (B) in the figure, after the user's second voice prompt to drive digital human 801 to speak, digital human 801's expression and posture changed in response to the voice message sent by the user. At this point, digital human 801's facial expression was enthusiastic and joyful (laughing), and its arms were raised above its head, its fists clenched, and its body slightly tilted to an encouraging gesture. This is also because the target prediction model predicts from the user's voice message that a real human would likely display this expression when speaking the voice message, for example, if the user's voice message contained content such as "Come on!"

[0202] Specifically, as described above, the target prediction model can predict the content from the user's voice information using facial expressions and torso posture parameters. The facial expressions can be the 52 key points corresponding to the face, which can locate specific facial features such as eyebrows, eyes, nose, and mouth to determine facial expressions. The torso posture parameters can be the 78 skeletal key points corresponding to the upper body, which can locate various parts of the upper body to determine the specific posture of the person.

[0203] from Figure 8 As can be seen in the figure, the target prediction model provided by this application adopts a fully convolutional network structure. Therefore, it can be deployed on mobile devices with low performance and computing power, such as mobile phones and tablets, through quantization methods using machine learning tools such as SNPE, QNN, TFLite, and coreML, effectively expanding the application scenarios of voice-driven models. In addition, because the target prediction model provided by this application can change the facial expressions of the digital human according to the voice sent by the user and simultaneously drive the digital human to make corresponding torso postures, it effectively improves the sensory realism of the digital human to the user.

[0204] Next, the driving method provided by the embodiment of the present application is introduced. Figure 3 After the target prediction model is obtained, the electronic device can drive the digital human to speak according to the driving method. Figure 9 As shown, the driving method includes but is not limited to the following steps:

[0205] S101. The electronic device obtains voice information.

[0206] The electronic device may be a mobile phone, a tablet computer, a wearable device, an in-vehicle device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc. This application does not impose any restrictions on the specific type of the electronic device. Specifically, the electronic device may be the electronic device 80 described above.

[0207] The above-mentioned voice information can be the voice collected by the above-mentioned electronic device after the user speaks in real time, or it can be a voice clip contained in a recording file, and this application does not limit this.

[0208] S102. The electronic device pre-processes the voice information to obtain a spectrogram corresponding to the voice information.

[0209] After collecting the above-mentioned voice information, the electronic device can use the audio pre-processing module (for example, Figure 3 The audio preprocessing module 301 in the embodiment of the present invention converts the speech signal into a spectrogram. The spectrogram can provide comprehensive information about the frequency, energy and time dimensions of the speech signal.

[0210] The spectrogram formation process includes multiple operations such as the audio preprocessing module pre-emphasizes the signal of the above-mentioned voice information, frames and windows it, performs short-time Fourier transform to obtain the spectrum of each frame signal, rotates and maps the spectrum, and splices the transformed multi-frame spectrum to form a spectrogram. For details, please refer to the above-mentioned relevant instructions and will not be repeated here.

[0211] In the embodiments of the present application, the spectrogram will serve as the input of the target prediction model provided by the present application.

[0212] S103. The electronic device processes the above-mentioned spectrogram using the target prediction model to obtain expression parameters and torso posture parameters corresponding to the above-mentioned voice information.

[0213] The target prediction model can extract and predict features of the spectrogram to obtain expression parameters and torso posture parameters corresponding to the voice information input by the user. The expression parameters and torso posture parameters can represent the facial expressions and body postures that a real human may make when speaking the above voice information through the mouth.

[0214] Specifically, in this application, the target prediction model is a fully convolutional neural network model, for example Figure 3 The target prediction model shown in . The target prediction model is a fully convolutional neural network model obtained by incompletely supervised training using target data. It can include an encoder and a decoder (including an expression decoder and a torso decoder). The encoder and decoder can be used to extract multiple features from the above-mentioned spectrogram. The encoder and decoder are both implemented based on a convolutional neural network architecture. In addition, the target model can also include a prediction module (including an expression prediction module and a torso posture prediction module) also implemented based on a convolutional neural network architecture. The prediction module can convolve the aforementioned extracted features and output the aforementioned expression parameters and torso posture parameters.

[0215] The above-mentioned target data includes three types of data: 1. Datasets with both expression labels and torso posture labels; 2. Datasets with only expression labels but no torso posture labels; 3. Datasets with only torso posture labels but no expression labels. Specifically, the present application can decouple the output head related to expression prediction from the output head related to torso posture prediction. During training, the datasets with both expression labels and torso posture labels and the data with only expression labels but no torso posture labels are used to train the output head related to expression prediction, while the datasets with both expression labels and torso posture labels and the data with only torso posture labels but no expression labels are used to train the output head related to torso posture prediction.

[0216] The specific architecture and related information of the above target prediction model can be referred to the Figure 3-Figure 7 The description is not repeated here.

[0217] S104. The electronic device uses the above-mentioned expression parameters and torso posture parameters to drive the digital human to speak and make corresponding gestures.

[0218] The electronic device can use the above-mentioned expression parameters and torso posture parameters to drive the digital human to speak with the predicted expression and make corresponding gestures.

[0219] Optionally, the digital human can be a cartoon digital human or a realistic digital human, which is not limited in this application.

[0220] The driving method provided in this application is implemented based on the target prediction model provided in this application. The target prediction model's network structure adopts a fully convolutional architecture, making it easier to deploy on mobile devices with lower computing power, such as mobile phones and tablets, effectively expanding the application scenarios of voice-driven models. Furthermore, because the target prediction model provided in this application can change the digital human's facial expressions based on the user's voice and simultaneously drive the digital human to make corresponding torso gestures, it effectively enhances the digital human's sensory realism for the user.

[0221] Next, the electronic device provided by the embodiments of the present application is introduced.

[0222] The electronic device is an electronic device that is deployed with the target prediction model provided by the present application, such as a mobile phone, a tablet computer, a wearable device, an in-vehicle device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA) or a special camera (such as a SLR camera, a card camera). Optionally, the electronic device 100 can be an electronic device under an ARM architecture or an electronic device under a non-ARM architecture. The present application does not impose any restrictions on the specific type of the electronic device. Specifically, it can be the electronic device 80 used in the aforementioned description, or it can be the aforementioned device for training the above-mentioned target prediction module.

[0223] Figure 10 The structure of the electronic device is exemplarily shown.

[0224] The electronic device 100 may include a processor 110, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone interface 170D, a display screen 194, etc.

[0225] It should be understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0226] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0227] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0228] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0229] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0230] The USB interface 130 is an interface that complies with USB standards and may be a Mini USB interface, a Micro USB interface, a USB Type-C interface, or the like. The USB interface 130 can be used to connect a charger to charge the electronic device 100, or to transfer data between the electronic device 100 and peripheral devices. It can also be used to connect headphones to play audio. This interface can also be used to connect other electronic devices, such as augmented reality devices.

[0231] It is understood that the interface connection relationship between the modules illustrated in the embodiment of the present invention is merely an illustrative illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0232] The charging management module 140 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also provide power to the electronic device via the power management module 141.

[0233] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 and provides power to the processor 110, the internal memory 121, the display 194, and the like. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be provided in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be provided in the same device.

[0234] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0235] Display screen 194 is used to display images, videos, and the like. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0236] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0237] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0238] The electronic device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0239] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0240] The speaker 170A, also called a "speaker", is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or listen to hands-free calls through the speaker 170A.

[0241] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or a voice message, the user can place the receiver 170B close to the ear to hear the voice.

[0242] Microphone 170C, also known as "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the source of sound, realize directional recording function, etc.

[0243] The headphone jack 170D is used to connect a wired headphone and can be the USB interface 130 or a 3.5mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0244] In some embodiments, the target prediction model provided in the present application for driving the digital human to speak and the voice information collected by the microphone 170C can be stored in the memory of the processor 110 or in the internal memory 121.

[0245] In some embodiments, after a user sends a voice message to the electronic device to drive the digital human to speak, the electronic device 100, under the control of the processor 110, can collect the voice message through the microphone 170C and input the voice message into the target prediction model deployed thereon. The target prediction model then outputs the expression parameters and torso posture parameters corresponding to the voice message. Furthermore, the processor 110 can play the collected voice message through the speaker 170A (the timbre of the original voice message can be adjusted, or background music can be added). Based on the expression parameters and torso posture parameters corresponding to the voice message output by the target prediction model, the facial expression and posture of the digital human displayed on the display screen 194 can be adjusted. This allows the user to perceive the digital human as speaking and flexibly switch expressions and postures according to the content of the speech.

[0246] An embodiment of the present application also provides an electronic device, which includes: one or more processors and a memory; wherein the memory is coupled to the one or more processors, and the memory is used to store computer program code, and the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method shown in the aforementioned embodiment.

[0247] As used in the above embodiments, the term “when…” may be interpreted to mean “if…” or “after…” or “in response to determining…” or “in response to detecting…”, depending on the context. Similarly, the phrases “upon determining…” or “if (stated condition or event) is detected” may be interpreted to mean “if determining…” or “in response to determining…” or “upon detecting (stated condition or event)” or “in response to detecting (stated condition or event)”, depending on the context.

[0248] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk).

[0249] Those skilled in the art will appreciate that all or part of the process steps in the above-described method embodiments can be implemented by a computer program instructing the relevant hardware. The program can be stored in a computer-readable storage medium, and when executed, the program can include the process steps in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A driving method, characterized in that: The method comprises: Preprocessing first voice information input by a user to obtain a first spectrogram corresponding to the first voice information; Processing the first spectrogram using a target prediction model to obtain target expression parameters and target torso posture parameters corresponding to the first speech information, wherein the target prediction model is a fully convolutional neural network model trained using target data, wherein the target data includes multiple pieces of data representing the correspondence between speech and facial expressions and / or body postures; Based on the target expression parameters and the target torso posture parameters, the digital human is driven to speak and make corresponding expressions and postures.

2. The method according to claim 1, characterized in that The target prediction model includes an encoder, an expression decoder, a torso decoder, an expression prediction module, and a torso posture prediction module. The encoder, the expression decoder, the torso decoder, the expression prediction module, and the torso posture prediction module are all convolutional neural networks. The target prediction model is used to process the first spectrogram to obtain target expression parameters and target torso posture parameters corresponding to the first speech information, including: Extracting features from the first spectrogram using the encoder to obtain a first feature; Decoding the first feature by the expression decoder and the torso decoder to obtain a second feature representing a correspondence between the first voice information and the facial expression and a third feature representing a correspondence between the first voice information and the human body posture; The second feature is processed by the expression prediction module to obtain the target expression parameter, and the third feature is processed by the torso posture prediction module to obtain the target torso posture parameter.

3. The method according to claim 2, characterized in that The encoder includes at least two encoding modules including a first encoding module, the expression decoder includes at least two expression decoding modules including a first expression decoding module, the torso decoder includes at least two torso decoding modules including a first torso decoder, the first encoding module is used to extract features from the first spectrogram to obtain a first encoding feature, the first encoding module is the first encoding module included in the encoder, the first expression decoding module is the last expression decoding module of the expression decoder, and the first torso decoding module is the last torso decoding module of the torso decoder. The expression decoder and the torso decoder respectively decode the first feature to obtain a second feature representing the correspondence between the first voice information and the facial expression and a third feature representing the correspondence between the first voice information and the human body posture, including: The first coding feature and the first expression decoding feature are processed by the first expression decoding module to obtain the second feature, and the first torso coding feature and the first expression decoding feature are processed by the first torso decoding module to obtain the third feature, the first expression decoding feature is the feature output by the previous expression decoding module of the first expression decoding module, and the first torso decoding feature is the feature output by the previous torso decoding module of the first torso decoding module.

4. The method according to any one of claims 1 to 3, characterized in that The target data includes at least one first data, at least one second data and at least one third data, the first data is data that contains both an expression label and a torso posture label, the second data is data that contains an expression label but does not contain a torso posture label, and the third data is data that contains a torso posture label but does not contain an expression label. The target prediction model is obtained by incompletely supervised training based on the target data.

5. The method according to claim 4, characterized in that The performing incomplete supervision training based on the target data includes: The encoder performs feature extraction on the second spectrogram to obtain a fourth feature; the expression decoder and the torso decoder respectively decode the fourth feature to obtain a fifth feature representing the correspondence between the second voice information and the facial expression, and a sixth feature representing the correspondence between the second voice information and the human body posture; the expression prediction module processes the fifth feature to obtain a first expression parameter, and the torso posture prediction module processes the sixth feature to obtain a first torso posture parameter, and the second spectrogram is obtained by the second voice information in the sample data; When the training data is the first data, the expression decoder is updated using the first expression parameters, and the encoder and the torso decoder are updated using the first torso posture parameters; when the training data is the second data, the encoder and the expression decoder are updated using the first expression parameters; when the training data is the third data, the encoder and the torso decoder are updated using the first torso posture parameters.

6. The method according to claim 4 or 5, characterized in that The target prediction model further includes an auxiliary expression prediction module and an auxiliary torso posture prediction module, and the method further includes: The auxiliary expression prediction module processes the fifth feature to obtain a seventh feature representing a correspondence between the second voice information and the facial expression, and the auxiliary torso posture prediction module processes the sixth feature to obtain an eighth feature representing a correspondence between the second voice information and the human body posture; The processing of the fifth feature by the expression prediction module to obtain the first expression parameter includes: When the sample data is the first data, processing the fifth feature and the eighth feature to obtain the first expression parameter; The processing of the sixth feature by the torso posture prediction module to obtain the first torso posture parameter includes: When the sample data is the first data, the fourth feature and the seventh feature are processed to obtain the first torso posture parameter.

7. The method according to claim 6, characterized in that The method further comprises: The auxiliary expression prediction module processes the fifth feature to obtain auxiliary expression parameters, and the auxiliary torso posture prediction module processes the sixth feature to obtain auxiliary torso posture parameters. When the sample data is the first data or the second data, the auxiliary expression prediction module, the expression decoder, and the encoder are updated using the auxiliary expression posture parameter; When the sample data is the first data or the third data, the auxiliary torso posture prediction module, the torso decoder and the encoder are updated using the auxiliary torso posture parameters.

8. An electronic device, characterized in that: The electronic device includes: one or more processors, a memory, and a display screen; The memory is coupled to the one or more processors, and is configured to store computer program codes, where the computer program codes include computer instructions. The one or more processors call the computer instructions to enable the electronic device to execute the method according to any one of claims 1 to 7.

9. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the processor is used to call computer instructions to enable the electronic device to execute the method as described in any one of claims 1-7.

10. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on an electronic device, the electronic device is caused to execute the method according to any one of claims 1 to 7.

Images