Facial action video generation method and device, readable medium and electronic device

Abstract

The present disclosure provides a facial action video generation method and device, a readable medium and an electronic device, and relates to the field of artificial intelligence technology. The method comprises: obtaining driving voice data; inputting the driving voice data into a facial action prediction model to obtain facial action driving parameters, wherein the facial action prediction model is obtained through a pre-training process; driving a facial model of a preset virtual object according to the facial action driving parameters to generate a facial action video corresponding to the driving voice data. The present disclosure can directly generate a facial action video according to the input driving voice data, thereby improving the accuracy and smoothness of facial expressions in the facial action video.

Description

Background Art

[0002] With the rapid development of science and technology, artificial intelligence (AI) has received more and more attention. With the introduction of digital virtual human technology, such as virtual anchors, virtual idols, and the Metaverse, various companies have launched their own virtual images. In order to ensure the authenticity of the virtual image, it is necessary to make the facial expressions of the virtual image when speaking fit the reality.

[0003] At present, the relevant virtual image facial expression driving scheme is to create a mapping relationship between different voice elements and facial driving parameters, and then determine the facial driving parameters based on the mapping relationship. However, this parameter mapping relationship that relies on multiple phonemes and facial templates can only drive the virtual person's mouth movement. The resulting facial movement video not only has low authenticity and accuracy, but also because the parameter mapping relationship between multiple phonemes and facial templates is not continuous, the changes in facial movements in the facial movement video may be sudden, and the smoothness of the facial movements is low. Summary of the invention

[0004] The purpose of the present disclosure is to provide a facial action video generation method, a facial action video generation device, a computer-readable medium and an electronic device, thereby improving the accuracy and authenticity of facial actions in facial action videos at least to a certain extent and ensuring the smoothness of facial actions.

[0005] According to a first aspect of the present disclosure, a method for generating a facial action video is provided, comprising:

[0006] Get driving voice data;

[0007] Inputting the driving speech data into a facial action prediction model to obtain facial action driving parameters, wherein the facial action prediction model is trained through a pre-training process;

[0008] The facial model of the preset virtual object is driven according to the facial action driving parameters to generate a facial action video corresponding to the driving voice data.

[0009] According to a second aspect of the present disclosure, there is provided a facial action video generating device, comprising:

[0010] A voice data acquisition module, used to acquire driving voice data;

[0011] A facial action driving parameter prediction module, used for inputting the driving voice data into a facial action prediction model to obtain facial action driving parameters, wherein the facial action prediction model is obtained through a pre-training process;

[0012] The facial action video generation module is used to drive the facial model of the preset virtual object according to the facial action driving parameters to generate the facial action video corresponding to the driving voice data.

[0013] According to a third aspect of the present disclosure, a computer-readable medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented.

[0014] According to a fourth aspect of the present disclosure, there is provided an electronic device, characterized in that it includes:

[0015] Processor; and

[0016] The memory is used to store one or more programs. When the one or more programs are executed by one or more processors, the one or more processors implement the above method.

[0017] A facial action video generation method provided by an embodiment of the present disclosure can obtain driving voice data, and then the driving voice data can be input into a facial action prediction model trained in a pre-training process to obtain facial action driving parameters, and then the facial model of a preset virtual object can be driven according to the facial action driving parameters to generate a facial action video corresponding to the driving voice data. The facial action driving parameters can be obtained by inputting the driving voice data into the facial action prediction model, which is not only simple to operate, but also compared with the method of determining the facial driving parameters by mapping the parameters of multiple phonemes and facial templates in the related art, the facial action driving parameters predicted by the facial action prediction model have better continuity, can ensure the smoothness of the facial actions in the facial action video, and can obtain the overall facial action, not only the lip action, so as to improve the accuracy and authenticity of the facial action.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0020] Figure 1 A schematic diagram showing an exemplary system architecture to which embodiments of the present disclosure may be applied;

[0021] Figure 2A schematic diagram schematically illustrates a flow chart of a facial action video generation method in an exemplary embodiment of the present disclosure;

[0022] Figure 3 A schematic diagram schematically illustrates a flow chart of a pre-training process corresponding to a facial action prediction model in an exemplary embodiment of the present disclosure;

[0023] Figure 4 A schematic diagram of a process of constructing audio features in an exemplary embodiment of the present disclosure is schematically shown;

[0024] Figure 5 A schematic diagram of a network structure of a facial action prediction model in an exemplary embodiment of the present disclosure is schematically shown;

[0025] Figure 6 A schematic diagram schematically illustrates the composition of a facial action video generating device in an exemplary embodiment of the present disclosure;

[0026] Figure 7 A schematic diagram of an electronic device to which the embodiments of the present disclosure can be applied is shown. DETAILED DESCRIPTION

[0027] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0028] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0029] Figure 1 A schematic diagram of a system architecture of an exemplary application environment in which a method and apparatus for generating a facial action video according to an embodiment of the present disclosure can be applied is shown.

[0030] like Figure 1As shown, the system architecture 100 may include one or more of terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc. The terminal devices 101, 102, 103 may be various electronic devices with image processing functions, including but not limited to desktop computers, portable computers, smart phones, tablet computers, etc. It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 105 may be a server cluster composed of multiple servers.

[0031] The facial action video generation method provided in the embodiment of the present disclosure is generally executed by the terminal devices 101, 102, and 103, and accordingly, the facial action video generation device is generally set in the terminal devices 101, 102, and 103. However, it is easy for those skilled in the art to understand that the facial action video generation method provided in the embodiment of the present disclosure can also be executed by the server 105, and accordingly, the facial action video generation device can also be set in the server 105, which is not particularly limited in this exemplary embodiment. For example, in an exemplary embodiment, the user can receive or collect driving voice data through the terminal devices 101, 102, and 103, and then upload the driving voice data to the server 105. After the server generates a facial action video through the facial action video generation method provided in the embodiment of the present disclosure, the facial action video is sent to the terminal devices 101, 102, and 103.

[0032] In a related technical solution, 2D video can be used to obtain the blended expression coefficient (BS) of the head model of the 3D model of facial expressions and shapes (Faces Learned with an Articulated Model and Expressions, FLAME) using the DECA 3D face reconstruction method, and a neural network is built to learn the mapping relationship from voice data to the Blendshape coefficient of FLAME. However, in this solution, the BS value of FLAME is obtained, and when driving digital virtual humans in rendering engines such as Unreal Engine (UE), Blendshape mostly uses the definition in ARKit (which is different from the definition of BS in FLAME); and if the BS value of FLAME is used, pytorch3D needs to be used to obtain the rendered images frame by frame, which has poor scalability.

[0033] In another related technical solution, the intensity value of the action unit (AU) of the facial action coding system (FACS) of each frame of the face image in the 2D video can be extracted based on the open source tool OpenFace, and the mapping relationship between the voice data and the AU value can be learned by using a neural network, and then the AU value can be converted to the mixed expression shape parameter BS value through an Unreal Engine such as UE4. However, to predict the AU value first and then convert it to the BS value through the AU value, it is necessary to build a facial action coding system for each frame of the face image, which is complicated to operate, and there is information loss during the conversion process, resulting in low accuracy of the generated facial actions.

[0034] In another related technical solution, the voice element corresponding to the input voice information can be obtained, and then according to the preset mapping relationship between the voice element and the facial driving parameter, the facial driving parameter is finally combined with the predetermined virtual human face template to render the virtual human face movement video in the Unreal Engine. However, this solution can only drive the virtual human mouth movement by relying solely on the parameter mapping relationship between multiple phonemes and facial templates, and the obtained facial movements have low accuracy, low smoothness, and low authenticity.

[0035] Based on one or more problems in the related art, the present disclosure first provides a facial action video generation method. The following takes a terminal device equipped with a facial action video generation device executing the method as an example to specifically illustrate the facial action video generation method of an exemplary embodiment of the present disclosure.

[0036] Figure 2 The flow chart of a facial action video generation method in this exemplary embodiment includes the following steps S210 to S230:

[0037] In step S210, driving voice data is acquired.

[0038] In an exemplary embodiment, driving voice data refers to voice data used to drive facial movements of a preset virtual object. For example, the driving voice data can be directly collected through an audio collection tool, or can be obtained by voice synthesis based on text data. Of course, it can also be extracted from a video, or obtained from other devices through communication methods. This exemplary embodiment does not impose any special restrictions on the source of the driving voice data.

[0039] In step S220, the driving speech data is input into a facial action prediction model to obtain facial action driving parameters, and the facial action prediction model is obtained through a pre-training process.

[0040] In an exemplary embodiment, the facial action prediction model refers to a deep learning model that can predict the facial action corresponding to the pronunciation of the voice data based on the voice data. For example, the facial action prediction model can be a deep learning model built based on convolutional neural networks (CNN) or a deep learning model built based on recurrent neural networks (RNN). This exemplary embodiment does not specifically limit the network type of the facial action prediction model.

[0041] Facial action driving parameters refer to driving parameters used to drive the facial model to achieve matching of facial actions with speech data. For example, the facial action driving parameters can be a mixed expression coefficient BS, or the displacement coordinates of multiple facial key points, or the intensity value of a motion unit of a facial action coding system. This example embodiment does not specifically limit the type of facial action driving parameters. The following description will take the facial action driving parameters as the mixed expression coefficient BS as an example.

[0042] The facial action prediction model is an end-to-end network structure. By inputting the driving voice data into the facial action prediction model, the facial action driving parameters corresponding to the driving voice data can be directly obtained. The operation is simple and the generation efficiency of the facial action driving parameters is improved.

[0043] In step S230, a facial model of a preset virtual object is driven according to the facial action driving parameters to generate a facial action video corresponding to the driving voice data.

[0044] In an exemplary embodiment, the preset virtual object refers to a pre-built three-dimensional model with a facial area. For example, the preset virtual object can be a humanoid virtual object restored based on facial key points. The facial area in the video or image can be detected, and multiple feature key points corresponding to the facial area can be collected, such as the 25 key points of the face, and three-dimensional modeling is performed through the 25 key points of the face to obtain a virtual object corresponding to the facial area in the video or image; of course, the preset virtual object can also be a cartoon image with a facial area. This exemplary embodiment does not impose any special limitation on the form of expression of the preset virtual object.

[0045] Optionally, the preset virtual object can be a three-dimensional model pre-modeled in a rendering engine. For example, the preset virtual object can be created in an Unreal Engine (such as UE4) or in a Unity engine. Of course, the preset virtual object can also be created in modeling software such as 3Ds max or Maya. This embodiment does not specifically limit the creation tool of the preset virtual object.

[0046] After obtaining the facial action driving parameters, the facial action driving parameters can be imported into the rendering engine or modeling tool corresponding to the preset virtual object, and the facial action driving parameters can be mapped to the facial model of the preset virtual object to drive the facial model to obtain continuous facial actions. The facial actions can be recorded through the virtual camera provided by the rendering engine or modeling tool to obtain the facial action video corresponding to the driving voice data.

[0047] Facial movement driving parameters can be obtained through the facial movement prediction model. Not only is the operation simple, but the facial movement driving parameters predicted by the facial movement prediction model have good continuity, which can ensure the smoothness of facial movements in facial movement videos and obtain the overall facial movement, not only the lip movement, thereby improving the accuracy and authenticity of facial movements.

[0048] In addition, the facial action video is obtained by driving the facial model of the preset virtual object in the rendering engine or modeling tool through facial action driving parameters. Compared with driving FLAME in the related technology and rendering frame by frame to obtain the facial action video, it has better scalability and a wider range of applications. The facial action changes in the obtained facial action video are smoother and more consistent with the facial actions in real scenes.

[0049] Steps S210 to S230 are described in detail below.

[0050] In an exemplary embodiment, the facial action prediction model can be trained through a pre-training process, and can be obtained by Figure 3 The steps in the pre-training process are implemented, refer to Figure 3 As shown, it may specifically include:

[0051] Step S310, acquiring training data, wherein the training data includes facial action driving parameters and audio sequences corresponding to the facial action driving parameters;

[0052] Step S320, preprocessing the audio sequence to obtain audio features;

[0053] Step S330, performing model training on the initial facial action prediction model using the facial action driving parameters and the audio features until the model loss of the initial facial action prediction model converges, thereby obtaining a trained facial action prediction model.

[0054] Among them, the initial facial action prediction model refers to a pre-constructed facial action prediction model with initial model parameters. The model parameters of the initial facial action prediction model can be iteratively updated through training data and the model loss function until the model loss of the initial facial action prediction model converges. At this time, the initial facial action prediction model with updated model parameters can be used as the trained facial action prediction model.

[0055] The training data of the initial facial action prediction model may include facial action driving parameters and an audio sequence corresponding to the facial action driving parameters. Specifically, the audio sequence can be used as input data of the initial facial action prediction model, and the facial action driving parameters can be used as label data of the initial facial action prediction model. The initial facial action prediction model is iteratively trained to finally obtain a trained facial action prediction model.

[0056] Optionally, training data can be obtained through the following steps, which may specifically include: collecting original data, which may include a video sequence of the demonstration subject when reading a test text and facial depth information of the demonstration subject when reading the test text, and determining facial action driving parameters based on the video picture and facial depth information in the video sequence; extracting an audio sequence corresponding to the facial action driving parameters from the video sequence.

[0057] Among them, raw data refers to unprocessed data collected by a collection tool. For example, the demonstration subject can be arranged in the shooting scene to read the pre-prepared text data loudly and clearly, and repeat the text data with a variety of emotions (such as happiness, sadness, etc.). The text data can be composed of multiple paragraphs of about 300 words, and each paragraph needs as many initials and finals as possible. Specifically, a collection tool, such as a smart phone, a camera, etc., can be set up so that the demonstration subject is facing the collection tool, and the face is in a preset area in the picture. The video sequence of the demonstration subject when reading the test text is collected, and the depth estimation can be performed based on the image frames in the video sequence to obtain the facial depth information of the demonstration subject when reading the test text. Optionally, the facial depth information of the demonstration subject when reading the test text can be obtained by a collection tool including a depth information collection device, and this example embodiment does not specifically limit this.

[0058] The video images and facial depth information in the video sequence can be aligned according to the timestamp information, and the facial action driving parameters can be extracted by superimposing the video images and facial depth information in the video sequence. Of course, the video sequence and facial depth information can also be input into a facial action capture tool to obtain the facial action driving parameters. For example, the facial action capture tool can be the Live Link Face application.

[0059] The audio sequence corresponding to the facial action driving parameters can be extracted from the video sequence by an audio extraction tool. For example, the audio sequence in the video sequence can be extracted by a video processing tool ffmpeg, and the audio sequence is aligned with the facial action driving parameters as training data. If the audio sequence and the facial action driving parameters are not aligned, the beginning and end parts of the extracted audio sequence can be truncated to ensure the alignment of the audio sequence and the facial action driving parameters.

[0060] By collecting the facial depth information of the demonstration subject when reading the test text, the facial action driving parameters are determined. Compared with the method of three-dimensional reconstruction based on 2D video or extraction of motion unit intensity values ​​in related technologies, this can effectively improve the accuracy of the facial action driving parameters in the training data and further ensure the accuracy of the generated facial action video.

[0061] In an exemplary embodiment, the Figure 4 The steps in the embodiment are to pre-process the audio sequence to obtain audio features, refer to Figure 4 As shown, it may specifically include:

[0062] Step S410, determining the Mel-frequency cepstral coefficient characteristics of the audio sequence;

[0063] Step S420, performing a first-order derivative on the Mel-frequency cepstral coefficient feature to obtain a first audio sequence feature;

[0064] Step S430, performing a second-order derivative on the Mel-frequency cepstral coefficient feature to obtain a second audio sequence feature;

[0065] Step S440: constructing an audio feature based on the Mel-frequency cepstral coefficient feature, the first audio sequence feature, and the second audio sequence feature.

[0066] The Mel-frequency cepstral coefficients (MFCC) feature refers to an audio feature obtained by linearly transforming the logarithmic energy spectrum of the nonlinear Mel scale of the sound frequency in the audio sequence.

[0067] The Mel-frequency cepstral coefficient feature can be first-order differentiated to obtain the first audio sequence feature, and the Mel-frequency cepstral coefficient feature can be second-order differentiated to obtain the second audio sequence feature, and then the multidimensional feature constructed by the Mel-frequency cepstral coefficient feature, the first audio sequence feature and the second audio sequence feature is used as the audio feature.

[0068] By inputting the Mel-frequency cepstral coefficient features, the first audio sequence features and the second audio sequence features as audio features into the initial facial action prediction model, the feature expression of the audio sequence can be effectively improved, the information loss of the encoded audio features can be avoided, and the accuracy of the facial action driving parameters can be improved.

[0069] Optionally, frame rate data of the facial action driving parameters may be determined, and then a sampling sliding window may be determined based on the frame rate data, and the audio features may be sampled based on the sampling sliding window to achieve alignment of the audio features with the facial action driving parameters.

[0070] Among them, the frame rate data refers to the collection frame rate of the facial movement driving parameters. For example, the collected facial movement driving parameters can be 30 frames / second or 60 frames / second. The frame rate data of the facial movement driving parameters is specifically related to the parameters of the collection tool. This example embodiment does not specifically limit the method for determining the frame rate data.

[0071] For example, taking the video collected by the facial motion capture tool Live link face as an example, the most stable frame rate is 30 frames per second, so 30 frames of facial motion driving parameters can be obtained every second. For example, the facial motion driving parameters can be BS parameters. Specifically, each frame of facial motion driving parameters can be composed of 61 BS parameters. The result of each video shot is a csv file with timestamp information and multiple frames of BS parameters. The collected original audio files can be pre-processed for noise reduction to eliminate ambient sounds and other noises and retain relatively pure human voices. Then, the denoised audio files can be extracted for MFCC features, first audio sequence features (deltas features), and second audio sequence features (deltas-deltas features).

[0072] Since the frame rate data of the facial action driving parameters is 30 frames per second, 0.02s and 1 / (30*4)s can be used as the size and step size of the sampling sliding window respectively. Every 4 segments of MFCC (1 / (30*4)*4) features correspond to one frame of BS parameters, so that the audio features and BS parameters are aligned in time sequence. Similarly, every 4 small segments of the first audio sequence features (deltas features) and the second audio sequence features (deltas-deltas feature extraction) constitute 1 second of audio features.

[0073] In an exemplary embodiment, the facial action prediction model may include an audio coding network, an audio coding network fully connected layer, a dropout layer, and a long short-term memory network.

[0074] Among them, the audio coding network can be a network layer built based on the CNN network, which can be used to abstract audio features and extract high-dimensional features corresponding to the audio features; through the audio coding network based on the CNN network with a simple and lightweight network structure, the high-dimensional features corresponding to the audio features can be quickly extracted, effectively improving the feature extraction efficiency.

[0075] The fully connected layer of the audio coding network can be composed of two fully connected layers and a ReLU activation layer, which is used to further abstract the audio features and output feature data with the same size as the input data of the long short-term memory network.

[0076] The dropout layer is used to discard the feature data output by the fully connected layer of the audio coding network to avoid overfitting of the fully connected layer of the audio coding network.

[0077] The Long Short-Term Memory (LSTM) network is mainly used to extract the temporal relationship in audio features, so that the output facial action driving parameters have good continuity and stability in time and space. Of course, although the Long Short-Term Memory network can learn temporal features, it does not highlight the importance of each feature between temporal features in the middle of the module, so it can also be replaced by a multi-head self-attention network (Multi-Head Self-Attention) structure to improve the algorithm effect.

[0078] Figure 5 A schematic diagram of a network structure of a facial action prediction model in an exemplary embodiment of the present disclosure is schematically shown.

[0079] refer to Figure 5 As shown, the facial action prediction model 500 may include an audio coding network 501 , an audio coding network fully connected layer 502 , a discard layer 503 , and a long short-term memory network 504 .

[0080] The audio coding network 501 can be used to abstract the Mel frequency cepstral coefficient features, the first audio sequence features and the second audio sequence features in the audio features again. Each sampling point of the Mel frequency cepstral coefficient feature is a one-dimensional vector with a length of 13, and the sampling rate is 16000HZ, that is, 16000 sampling points per second. Then use 0.02s as the window length and 1 / (30*4) as the step length of the sampling sliding window combination feature, and 4 audio features constitute 1 second of features (the first audio sequence features and the second audio sequence are the same). Here, it is necessary to process 3 13*N (in the actual network, 12*28 features are processed, and the features of the first dimension are discarded) two-dimensional features (Mel frequency cepstral coefficient features, first audio sequence features and second audio sequence features), and an audio spectrum feature map of 3 channels * width 13 * length N can be abstracted.

[0081] Since the BS parameters are correlated in time series, in order to solve the BS parameters corresponding to the t frame, the 28 audio features [(t-3)*4, (t+4)*4] are used as a group to solve. The CNN network is good at extracting image features. The audio coding network 501 is designed as a CNN network to encode the spectral features of each frame; the audio coding network fully connected layer 502 is used for further abstraction, followed by a dropout layer 503 to prevent the audio coding network fully connected layer 502 from overfitting. The temporal extraction of audio features uses a long short-term memory network 504. The decoded audio features are grouped into 16 frames and input into the long short-term memory network 504 to learn the relationship between the features of these 16 frames; the feature data obtained by the long short-term memory network 504 is mapped to 61 ARkit BS parameters using full connection as the final facial action driving parameter output.

[0082] The network structure of the audio coding network 501 may be as shown in Table 1:

[0083] Table 1 Network structure of audio coding network

[0084] Network layer name Output data size Audio Codec Conv1 28*12 3*1,64,stride=1 Conv2 28*12 3*1,128, stride=1 Maxpool1 28*6 3, stride = (1, 2) Conv3 28*6 3*1,256,stride=1 Conv4 28*6 3*1,256,stride=1 Conv5 28*6 3*1,512, stride=1 Maxpool2 12*3 3, stride = (2, 2)

[0085] The audio coding network fully connected layer 502 is composed of two fully connected layers and a ReLU activation layer, and finally obtains a one-dimensional feature of length 256. The 16 one-dimensional features of length 256 are passed through the long short-term memory network 504 to obtain 16*256 feature data, which is finally mapped to 16*61 BS parameters by full connection.

[0086] The facial action prediction model 500 is trained in an end-to-end manner, and the loss function can be obtained by weighting two parts. The first part is the Huber loss for determining the accuracy of the output facial action driving parameters; the second part is the loss for determining the continuity of the output facial action driving parameters, which is used to eliminate unreasonable mutation points between frames of the output facial action driving parameters to ensure continuity. The specific loss function can be expressed by the relationship (1):

[0087]

[0088] Among them, Huberloss can be expressed by equation (2):

[0089]

[0090] Among them, β can represent the facial action driving parameters as label data, that is, the 61 BS parameters in the training data, λ may represent 61 BS parameters predicted by the facial action prediction model 500, and λ and δ may represent hyperparameters corresponding to the facial action prediction model 500. Of course, this is only an illustrative example and does not constitute any special limitation to this exemplary embodiment.

[0091] In summary, in this exemplary embodiment, driving voice data can be obtained, and then the driving voice data can be input into the facial action prediction model obtained by the pre-training process to obtain facial action driving parameters, and then the facial model of the preset virtual object can be driven according to the facial action driving parameters to generate a facial action video corresponding to the driving voice data. The facial action driving parameters can be obtained by inputting the driving voice data into the facial action prediction model, which is not only simple to operate, but also compared with the method of determining the facial driving parameters by mapping the parameters of multiple phonemes and facial templates in the related art, the facial action driving parameters predicted by the facial action prediction model have better continuity, can ensure the smoothness of facial actions in the facial action video, and can obtain the overall facial action, not only the lip action, so as to improve the accuracy and authenticity of the facial action.

[0092] The facial action prediction model in the disclosed embodiment is an end-to-end lightweight network design. The audio signal of the input speech, that is, the driving speech data, can quickly output the facial action driving parameters, such as the coefficients of 61 BS, through the designed facial action prediction model. In various virtual engines such as Unity, it can very easily drive the face of the virtual object to move. It is applicable to any language as input data, and the corresponding facial action driving parameters can be learned. And this method is applicable to any scene, such as the scene of various interactions through virtual people, and the facial action video of the virtual person can be generated very quickly.

[0093] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0094] For further reference, Figure 6 As shown, in the embodiment of this example, a facial action video generation device 600 is also provided, including a voice data acquisition module 610, a facial action driving parameter prediction module 620 and a facial action video generation module 630. Among them:

[0095] The voice data acquisition module 610 is used to acquire driving voice data;

[0096] The facial action driving parameter prediction module 620 is used to input the driving voice data into a facial action prediction model to obtain facial action driving parameters, wherein the facial action prediction model is obtained through a pre-training process;

[0097] The facial action video generation module 630 is used to drive the facial model of the preset virtual object according to the facial action driving parameters to generate the facial action video corresponding to the driving voice data.

[0098] In an exemplary embodiment, the pre-training process of the facial action prediction model may include:

[0099] A training data acquisition unit, used to acquire training data, wherein the training data includes facial action driving parameters and audio sequences corresponding to the facial action driving parameters;

[0100] An audio preprocessing unit, used for preprocessing the audio sequence to obtain audio features;

[0101] A model training unit is used to perform model training on the initial facial action prediction model through the facial action driving parameters and the audio features until the model loss of the initial facial action prediction model converges, thereby obtaining a trained facial action prediction model.

[0102] In an exemplary embodiment, the training data acquisition unit may be used to:

[0103] Collecting raw data, the raw data including a video sequence of a demonstration subject when reading a test text and facial depth information of the demonstration subject when reading the test text;

[0104] Determine facial action driving parameters according to the video images in the video sequence and the facial depth information;

[0105] An audio sequence corresponding to the facial action driving parameters is extracted from the video sequence.

[0106] In an exemplary embodiment, the audio pre-processing unit may be configured to:

[0107] Determining Mel-frequency cepstral coefficient characteristics of the audio sequence;

[0108] Taking a first-order derivative of the Mel-frequency cepstral coefficient feature to obtain a first audio sequence feature;

[0109] Taking a second-order derivative of the Mel-frequency cepstral coefficient feature to obtain a second audio sequence feature;

[0110] An audio feature is constructed based on the Mel-frequency cepstral coefficient feature, the first audio sequence feature, and the second audio sequence feature.

[0111] In an exemplary embodiment, the audio pre-processing unit may be configured to:

[0112] Determining frame rate data of the facial action driving parameters;

[0113] A sampling sliding window is determined according to the frame rate data, and the audio features are sampled according to the sampling sliding window to achieve alignment of the audio features with the facial action driving parameters.

[0114] In an exemplary embodiment, the facial action prediction model may include an audio coding network, an audio coding network fully connected layer, a dropout layer, and a long short-term memory network.

[0115] In an exemplary embodiment, the facial action driving parameters may include mixed expression coefficients.

[0116] The specific details of each module in the above device have been described in detail in the implementation method of the method part. The undisclosed details can be found in the implementation method of the method part, so they will not be repeated here.

[0117] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0118] An exemplary embodiment of the present disclosure provides an electronic device for implementing a facial action video generation method, which may be Figure 1 The terminal device 101, 102, 103 or the server 105 in the embodiment of the present invention comprises at least a processor and a memory, the memory is used to store executable instructions of the processor, and the processor is configured to execute the facial action video generation method by executing the executable instructions.

[0119] Below Figure 7 Taking the electronic device 700 in FIG. 1 as an example, the structure of the electronic device in the present disclosure is exemplarily described. Figure 7 The electronic device 700 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0120] like Figure 7 As shown, the electronic device 700 is in the form of a general computing device. The components of the electronic device 700 may include, but are not limited to: at least one processing unit 710, at least one storage unit 720, a bus 730 connecting different system components (including the storage unit 720 and the processing unit 710), and a display unit 740.

[0121] The storage unit 720 stores program codes, and the program codes can be executed by the processing unit 710, so that the processing unit 710 executes the facial action video generation method in this specification.

[0122] The storage unit 720 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 721 and / or a cache memory unit 722 , and may further include a read-only memory unit (ROM) 723 .

[0123] The storage unit 720 may also include a program / utility 724 having a set (at least one) of program modules 725, such program modules 725 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0124] Bus 730 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0125] The electronic device 700 may also communicate with one or more external devices 770 (e.g., sensor devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 700, and / or communicate with any device that enables the electronic device 700 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 750. In addition, the electronic device 700 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 760. As shown, the network adapter 760 communicates with other modules of the electronic device 700 via a bus 730. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data backup storage systems, and sensor modules (e.g., gyroscope sensors, magnetic sensors, acceleration sensors, distance sensors, proximity light sensors, etc.).

[0126] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiment of the present disclosure.

[0127] The exemplary embodiments of the present disclosure also provide a computer-readable storage medium on which a program product capable of implementing the above-mentioned method of the present specification is stored. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of the present specification.

[0128] It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0129] In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0130] In addition, program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0131] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0132] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A facial action video generation method, characterized in that: include: Get driving voice data; Inputting the driving speech data into a facial action prediction model to obtain facial action driving parameters, wherein the facial action prediction model is obtained through a pre-training process; Driving a facial model of a preset virtual object according to the facial action driving parameters to generate a facial action video corresponding to the driving voice data; Wherein, the pre-training process includes: Collecting raw data, the raw data including a video sequence of a demonstration subject when reading a test text and facial depth information of the demonstration subject when reading the test text; Aligning the video images and facial depth information in the video sequence according to the timestamp information, and extracting facial action driving parameters by superimposing the video images and facial depth information in the video sequence; Extracting an audio sequence corresponding to the facial action driving parameters from the video sequence; The training data includes facial action driving parameters and audio sequences corresponding to the facial action driving parameters; Preprocessing the audio sequence to obtain audio features; The initial facial action prediction model is trained by using the facial action driving parameters and the audio features until the model loss of the initial facial action prediction model converges, thereby obtaining a trained facial action prediction model.

2. The method according to claim 1, characterized in that: The preprocessing of the audio sequence to obtain the audio features includes: Determining Mel-frequency cepstral coefficient characteristics of the audio sequence; Taking a first-order derivative of the Mel-frequency cepstral coefficient feature to obtain a first audio sequence feature; Taking a second-order derivative of the Mel-frequency cepstral coefficient feature to obtain a second audio sequence feature; An audio feature is constructed based on the Mel-frequency cepstral coefficient feature, the first audio sequence feature, and the second audio sequence feature.

3. The method according to claim 2, characterized in that The preprocessing of the audio sequence to obtain the audio features includes: Determining frame rate data of the facial action driving parameters; A sampling sliding window is determined according to the frame rate data, and the audio features are sampled according to the sampling sliding window to achieve alignment of the audio features with the facial action driving parameters.

4. The method according to claim 1, characterized in that: The facial action prediction model includes an audio coding network, an audio coding network fully connected layer, a dropout layer, and a long short-term memory network.

5. The method according to claim 1, characterized in that The facial action driving parameters include mixed expression coefficients.

6. A facial action video generation device, characterized in that: include: A voice data acquisition module, used to acquire driving voice data; A facial action driving parameter prediction module, used for inputting the driving voice data into a facial action prediction model to obtain facial action driving parameters, wherein the facial action prediction model is obtained through a pre-training process; A facial action video generation module, used to drive a facial model of a preset virtual object according to the facial action driving parameters, and generate a facial action video corresponding to the driving voice data; The pre-training process performed by the device includes: Collecting raw data, the raw data including a video sequence of a demonstration subject when reading a test text and facial depth information of the demonstration subject when reading the test text; Aligning the video images and facial depth information in the video sequence according to the timestamp information, and extracting facial action driving parameters by superimposing the video images and facial depth information in the video sequence; Extracting an audio sequence corresponding to the facial action driving parameters from the video sequence; The training data includes facial action driving parameters and audio sequences corresponding to the facial action driving parameters; Preprocessing the audio sequence to obtain audio features; The initial facial action prediction model is trained by using the facial action driving parameters and the audio features until the model loss of the initial facial action prediction model converges, thereby obtaining a trained facial action prediction model.

7. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

8. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 5 by executing the executable instructions.

Images