Method, device and equipment for generating audio-driven three-dimensional facial animation model
By extracting and encoding features from sample data, and using audio-driven 3D facial animation models to generate high-precision hybrid deformation values, the problem of mismatch between lip shape and speech in voice-driven facial expression technology is solved, achieving high-precision lip shape and facial expression rendering, which is suitable for film and television production.
Patent Information
- Application Number
- CN202311870903.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-29
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2043-12-29
AI Technical Summary
In existing voice-driven facial expression technology, lip movements do not correspond to speech, and facial expressions lack naturalness, resulting in low accuracy of audio-driven 3D facial animation technology.
By acquiring sample data for feature extraction and encoding, the audio to be trained drives the 3D face animation model for convolution and encoding, calculates the loss function value and updates the model parameters, generates high-precision hybrid deformation values, and combines the preset Unreal Engine to achieve high-precision rendering of lip movements and facial expressions.
It improves the accuracy of audio-driven 3D facial animation technology, achieving high-precision reproduction of lip movements and facial expressions, and is suitable for scenarios with high requirements for expression accuracy, such as film and television works.
Smart Images

Figure CN117710541B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of audio processing technology, and in particular to a method, device and equipment for generating an audio-driven three-dimensional facial animation model. Background Art
[0002] The research on voice-driven expression technology is an important part of the field of natural human-computer interaction. Specifically, it is to drive the digital image through voice to express the corresponding voice content with the facial expression corresponding to the voice, and then generate the corresponding animation. The core of voice-driven expression technology lies in the calculation and output of blended shape values (BlendShapes values, referred to as BS values). Specifically, the user's recorded or TTS synthesized voice is preprocessed and the BS value is output. Based on the BS value, it can be used to drive the digital image to generate the facial expression animation corresponding to the voice, and then it is presented on various display devices through rendering technology.
[0003] In the related art, in the production process of animation, film and television works, it is often necessary to modify the lines later. However, the asynchronism between audio and image caused by modifying the lines will seriously reduce the effect of the film and television works. Therefore, the related art processes it by reshooting, or by post-production and rendering, but the above methods are all very expensive. In order to overcome the above problems, the related art realizes the modification of lines through audio-driven 3D facial animation technology, that is, the original image is driven by the audio corresponding to the modified lines to generate the facial expression animation corresponding to the modified lines.
[0004] However, the BS value output by the above-mentioned voice-driven expression technology has a series of problems, such as the lip movements not corresponding to the voice, and the naturalness of the expression movements is not good, which reduces the accuracy of the audio-driven 3D facial animation technology. Summary of the invention
[0005] In order to solve the above technical problems, the embodiments of the present application provide a method, device and apparatus for generating an audio-driven 3D facial animation model, which can improve the accuracy of audio-driven 3D facial animation technology. The technical solution is as follows:
[0006] According to a first aspect of an embodiment of the present application, a method for generating an audio-driven three-dimensional facial animation model is provided, the method comprising:
[0007] Acquire sample data, the sample data including sample audio data, sample speaking style data and sample mixed deformation value, the sample audio data and the sample speaking style data belong to the same user, the sample speaking style data is used to describe the user's facial expression, and the sample mixed deformation value is obtained by preprocessing the sample audio data;
[0008] Performing feature extraction on the sample audio data to obtain sample audio features;
[0009] Convolving the sample audio features based on the audio to be trained driving the three-dimensional facial animation model to obtain initial audio features; and encoding the sample speech style data based on the audio to be trained driving the three-dimensional facial animation model to obtain sample speech style features;
[0010] Encoding the initial audio features and the speaking style features of the sample based on the audio to be trained to drive the three-dimensional facial animation model, and obtaining an output mixed deformation value;
[0011] Calculate the sample mixed deformation value and the output mixed deformation value to obtain a loss function value;
[0012] The model parameters of the audio-driven three-dimensional facial animation model to be trained are updated based on the loss function value.
[0013] In a possible implementation manner of the first aspect, extracting features from the sample audio data to obtain sample audio features includes:
[0014] Feature extraction is performed on the sample audio data based on a preset model, and the features of the middle layer of the preset model are used as the sample audio features.
[0015] In a possible implementation of the first aspect, the driving of the three-dimensional facial animation model based on the audio to be trained to perform convolution on the sample audio feature to obtain the initial audio feature includes:
[0016] Convolving the sample audio feature based on the audio to be trained to drive the three-dimensional facial animation model to obtain at least one intermediate audio feature;
[0017] Perform interval calculation on the at least one intermediate audio feature to obtain the initial audio feature.
[0018] In a possible implementation manner of the first aspect, performing interval calculation on the at least one intermediate audio feature to obtain the initial audio feature includes:
[0019] Based on the audio to be trained, the three-dimensional facial animation model is driven to match every two intermediate audio features in the at least one intermediate audio feature to obtain an intermediate audio feature group corresponding to every two intermediate audio features in the at least one intermediate audio feature; wherein each intermediate audio feature in the at least one intermediate audio feature corresponds to a convolution calculation channel, and sequence values of the two convolution calculation channels corresponding to the intermediate audio feature group are not adjacent;
[0020] Based on the audio to be trained, the three-dimensional facial animation model is driven to merge two intermediate audio features in each intermediate audio feature group in at least one intermediate audio feature group to obtain an intermediate merged feature corresponding to each intermediate audio feature group in the at least one intermediate audio feature group;
[0021] Based on the audio to be trained, the three-dimensional facial animation model is driven to calculate each intermediate merged feature in at least one intermediate merged feature to obtain the initial audio feature.
[0022] In a possible implementation of the first aspect, the three-dimensional facial animation model driven by the audio to be trained is encoded with the sample speech style data to obtain the sample speech style feature, including:
[0023] The sample speech style data is one-hot encoded based on the audio-driven three-dimensional facial animation model to be trained to obtain the sample speech style features.
[0024] In a possible implementation of the first aspect, the three-dimensional facial animation model driven by the audio to be trained is encoded with the initial audio feature and the sample speech style feature to obtain an output mixed deformation value, including:
[0025] Based on the audio to be trained, the three-dimensional facial animation model is driven to superimpose the initial audio feature and the sample speaking style feature to obtain a sample superimposed feature;
[0026] The sample superposition feature is encoded based on the audio-driven three-dimensional human face animation model to be trained to obtain the output mixed deformation value.
[0027] In a possible implementation manner of the first aspect, encoding the sample superposition feature based on the audio-driven three-dimensional human face animation model to be trained to obtain the output blend deformation value includes:
[0028] Encoding the sample superimposed features based on the audio-driven three-dimensional facial animation model to be trained to obtain sample encoded features;
[0029] The sample encoding feature is decoded based on the audio-driven three-dimensional facial animation model to be trained to obtain an output blending deformation value.
[0030] According to a second aspect of an embodiment of the present application, there is provided a device for generating an audio-driven three-dimensional facial animation model, the device comprising:
[0031] An acquisition module is used to acquire sample data, the sample data including sample audio data, sample speech style data and sample mixed deformation value, the sample audio data and the sample speech style data belong to the same user, the sample speech style data is used to describe the user's facial expression, and the sample mixed deformation value is obtained by preprocessing the sample audio data;
[0032] A feature extraction module is used to extract features from the sample audio data to obtain sample audio features;
[0033] The first training module is used to drive the three-dimensional facial animation model to convolve the sample audio features based on the audio to be trained to obtain initial audio features; and to drive the three-dimensional facial animation model to encode the sample speech style data based on the audio to be trained to obtain sample speech style features;
[0034] A second training module is used to drive the three-dimensional facial animation model to encode the initial audio features and the sample speaking style features based on the audio to be trained to obtain an output mixed deformation value;
[0035] A calculation module, used to calculate the sample mixed deformation value and the output mixed deformation value to obtain a loss function value;
[0036] An updating module is used to update the model parameters of the audio-driven three-dimensional facial animation model to be trained based on the loss function value.
[0037] In a possible implementation manner of the second aspect, the feature extraction module includes:
[0038] The feature extraction unit is used to extract features from the sample audio data based on a preset model, and use the features of the middle layer of the preset model as the sample audio features.
[0039] In a possible implementation manner of the second aspect, the first training module includes:
[0040] A convolution unit, configured to drive the three-dimensional face animation model to perform convolution on the sample audio feature based on the audio to be trained, so as to obtain at least one intermediate audio feature;
[0041] The interval calculation unit is used to perform interval calculation on the at least one intermediate audio feature to obtain the initial audio feature.
[0042] In a possible implementation manner of the second aspect, the interval calculation unit includes:
[0043] A first matching subunit is used to drive the three-dimensional facial animation model to match every two intermediate audio features of the at least one intermediate audio feature based on the audio to be trained, so as to obtain an intermediate audio feature group corresponding to every two intermediate audio features of the at least one intermediate audio feature; wherein each intermediate audio feature of the at least one intermediate audio feature corresponds to a convolution calculation channel, and sequence values of the two convolution calculation channels corresponding to the intermediate audio feature group are not adjacent;
[0044] a merging subunit, configured to drive the three-dimensional facial animation model to merge two intermediate audio features in each intermediate audio feature group in at least one intermediate audio feature group based on the audio to be trained, so as to obtain an intermediate merged feature corresponding to each intermediate audio feature group in the at least one intermediate audio feature group;
[0045] The calculation subunit is used to drive the three-dimensional facial animation model to calculate each intermediate merged feature in at least one intermediate merged feature based on the audio to be trained to obtain the initial audio feature.
[0046] In a possible implementation manner of the second aspect, the first training module includes:
[0047] The one-hot encoding module is used to perform one-hot encoding on the sample speech style data based on the audio-driven three-dimensional human face animation model to be trained, so as to obtain the sample speech style features.
[0048] In a possible implementation manner of the second aspect, the second training module includes:
[0049] A superposition unit, used for driving a three-dimensional facial animation model based on the audio to be trained to superimpose the initial audio feature and the sample speaking style feature to obtain a sample superposition feature;
[0050] The encoding unit is used to encode the sample superposition feature based on the audio-driven three-dimensional human face animation model to be trained to obtain the output mixed deformation value.
[0051] In a possible implementation manner of the second aspect, the encoding unit includes:
[0052] An encoding subunit, used for encoding the sample superimposed features based on the audio-driven three-dimensional human face animation model to be trained to obtain sample encoding features;
[0053] The decoding subunit is used to decode the sample encoding features based on the audio to be trained to drive the three-dimensional facial animation model to obtain an output mixed deformation value.
[0054] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a processor and a memory, the memory being used to store at least one program, the at least one program being loaded by the processor and executing a method for generating an audio-driven three-dimensional facial animation model as provided in the first aspect above.
[0055] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, characterized in that at least one program is stored in the computer-readable storage medium, and the at least one program is loaded and executed by a processor to implement the method for generating an audio-driven three-dimensional facial animation model as provided in the first aspect above.
[0056] According to a fifth aspect of an embodiment of the present application, a computer program product comprising computer instructions is provided. When the computer instructions are executed on an electronic device, the electronic device implements the method for generating an audio-driven three-dimensional facial animation model as provided in the first aspect above.
[0057] In this application, the above names do not limit the devices or functional modules themselves. In actual implementation, these devices or functional modules may appear with other names. As long as the functions of each device or functional module are similar to those of this application, they fall within the scope of the claims of this application and their equivalent technologies.
[0058] These and other aspects of the present application will become more apparent from the following description.
[0059] In an embodiment of the present application, an embodiment of the present application provides a method for generating an audio-driven three-dimensional facial animation model, obtaining sample data of a user, the sample data including sample audio data, sample speech style data, and sample blending deformation values corresponding to the sample audio data, wherein the sample speech style data describes the facial expression data of the user, so as to fuse the personalized data of the user with the sample audio data, and retain the personalized data of the user as much as possible. Based on the sample data, the audio-driven three-dimensional facial animation model to be trained is trained, and the trained audio-driven three-dimensional facial animation model obtained can output a high-precision BS value, thereby improving the accuracy of the audio-driven three-dimensional facial animation technology. In addition, the output high-precision BS value is input into a preset Unreal Engine, so that the preset Unreal Engine can render high-precision lip shape and facial expression to a video display device, thereby simultaneously realizing high-precision reproduction of lip shape and facial expression, and then the audio-driven three-dimensional facial animation technology can be widely used in scenes with high expression accuracy requirements such as film and television production. BRIEF DESCRIPTION OF THE DRAWINGS
[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0061] Figure 1 is a schematic diagram of an implementation environment provided according to an embodiment of the present application;
[0062] Figure 2A It is a flowchart of a method for generating an audio-driven three-dimensional facial animation model according to an embodiment of the present application;
[0063] Figure 2B This is a first exemplary flow chart of a method for generating an audio-driven three-dimensional facial animation model according to an embodiment of the present application;
[0064] Figure 2C is a second example flow chart of a method for generating an audio-driven three-dimensional facial animation model provided in an embodiment of the present application;
[0065] Figure 2D is a third exemplary flow chart of a method for generating an audio-driven three-dimensional facial animation model provided in an embodiment of the present application;
[0066] Figure 2E is a fourth exemplary flow chart of a method for generating an audio-driven three-dimensional facial animation model provided in an embodiment of the present application;
[0067] Figure 2F is a fifth exemplary flow chart of a method for generating an audio-driven three-dimensional facial animation model provided in an embodiment of the present application;
[0068] Figure 3 It is a structural schematic diagram of a device for generating an audio-driven three-dimensional facial animation model according to an embodiment of the present application;
[0069] Figure 4 is a schematic diagram of the structure of a terminal provided according to an embodiment of the present application;
[0070] Figure 5 It is a structural diagram of a server provided according to an embodiment of the present application. DETAILED DESCRIPTION
[0071] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.
[0072] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0073] In this application, the terms "first", "second", etc. are used to distinguish between identical or similar items with substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is there a limitation on the quantity and execution order. It should also be understood that although the following description uses the terms first, second, etc. to describe various elements, these elements should not be limited by the terms.
[0074] These terms are only used to distinguish one element from another element. For example, without departing from the scope of various examples, a first action can be referred to as a second action, and similarly, a second action can also be referred to as a first action. Both the first action and the second action can be actions, and in some cases, can be separate and different actions.
[0075] Here, at least one means one or more than one, for example, at least one action can be one action, two actions, three actions, or any other action that is an integer greater than or equal to one. And multiple means two or more than two, for example, multiple actions can be two actions, three actions, or any other action that is an integer greater than or equal to two.
[0076] It should be noted that the data (including but not limited to training data and prediction data, such as user data, terminal side data, etc.) and signals involved in this application are authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.
[0077] Figure 1 It is a schematic diagram of an implementation environment provided according to an embodiment of the present application, and the implementation environment may include a terminal 101 and a server 102.
[0078] In the terminal 101, an audio receiving device and a video display device are provided. The audio receiving device and the video display device may be independent devices or integrated into a hardware device, which integrates a sound receiving function and an animation display function, such as an LED (Light Emitting Diode) screen with a voice recognition function. The terminal 101 may be a wearable device, a personal computer, a laptop, a tablet computer, a smart TV, a vehicle-mounted terminal, etc.
[0079] The server 102 may be a single server, a server cluster consisting of multiple servers, or a cloud processing center.
[0080] The terminal 101 is connected to the server 102 via a wired or wireless network.
[0081] In some embodiments, the wireless network or wired network uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to any combination of local area network (LAN), metropolitan area network (MAN), wide area network (MAN), mobile, wired or wireless network, private network or virtual private network. In some embodiments, the data exchanged through the network is represented by technology and / or format including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technology can also be used to replace or supplement the above data communication technology.
[0082] Figure 2A is a flow chart of a method for generating an audio-driven three-dimensional facial animation model according to an embodiment of the present application, such as Figure 2A As shown, in the embodiment of the present application, an example of application is given with an audio receiving device and a video display device. The method comprises the following steps:
[0083] In step 201, the terminal obtains sample data, which includes sample audio data, sample speaking style data and sample mixed deformation values. The sample audio data and the sample speaking style data belong to the same user. The sample speaking style data is used to describe the user's facial expression. The sample mixed deformation value is obtained by preprocessing the sample audio data.
[0084] It should be noted that, in some embodiments, the blending deformation value may also be expressed as a BS value.
[0085] In some embodiments, Figure 2B As shown, the above step 201 includes the following steps:
[0086] In step 2011, the terminal obtains initial data.
[0087] In some embodiments, at least one fixed speech sample is preset (for example, 100), and the at least one fixed speech sample is assigned to at least one user, so that each of the at least one user faces a data acquisition device in the same environment, reads the speech sample assigned to him or her, and expresses a preset facial expression, thereby obtaining initial data through the data acquisition device.
[0088] In some embodiments, the terminal performs real-time facial capture of each of at least one user based on the structured light in the data acquisition device and the built-in augmented reality development platform, and obtains at least one frame of image corresponding to each of at least one user, wherein each frame of the at least one frame of image is used to describe the facial expression of the corresponding user, and simultaneously obtains and records the audio corresponding to each frame of the at least one frame of image and at least one BS value. Among them, each frame of the at least one frame of image, the audio corresponding to each frame of the at least one frame of image, and at least one BS value are used as an initial data, that is, at least one initial data can be obtained through the above-mentioned data acquisition device. The embodiments of the present application do not specifically limit the data acquisition device.
[0089] Optionally, any one of the at least one BS value can be obtained in the following manner: by mixing the audio to obtain a relative proportion or weight between the audio signals, wherein the relative proportion or weight is a BS value.
[0090] It should be noted that the audio may be mixed based on a conventional algorithm to obtain a BS value corresponding to the audio, and the embodiments of the present application do not elaborate on the specific process of the mixing process.
[0091] In step 2012, the terminal screens the initial data at least once to obtain screened data.
[0092] In some embodiments, the initial data obtained above is screened according to preset parameters such as environmental errors and / or user errors, and at least one frame of image in which user errors and / or environmental errors occur is deleted. At the same time, the audio and BS value corresponding to the at least one frame of image are also deleted to obtain filtered data with better quality.
[0093] In step 2013, the terminal optimizes the screening data to obtain sample data.
[0094] In some embodiments, images that inaccurately express the facial expressions of some users are adjusted through animation production to optimize the images that inaccurately express the facial expressions of the users, thereby obtaining sample facial expression data that more accurately expresses the facial expressions, and obtaining sample speaking style data based on the sample facial expression data, thereby obtaining more accurate sample data.
[0095] Optionally, when obtaining sample speaking style data based on sample facial expression data, taking a user as an example, each frame of at least one frame of image describing the sample facial expression data of the user is judged to determine the speaking style of the user, obtain the speaking style data of the user, and then retain the sample facial expression data of the user as much as possible. For example, the speaking style data includes: "exaggerated" and "gentle", etc.
[0096] In step 202, the terminal extracts features from the sample audio data to obtain sample audio features.
[0097] In some embodiments, the above step 202 includes: the terminal extracts features from the sample audio data based on a preset model, and uses the features of the middle layer of the preset model as the sample audio features. Optionally, the preset model includes an input layer, an middle layer, and an output layer. The middle layer includes multiple data processing sublayers. Optionally, when obtaining the sample audio features, the features of a data processing sublayer in the middle layer can be used as the sample audio features.
[0098] Optionally, since the audio receiving devices used for data collection from users and the audio sources of users are various, there is at least one audio data format in the collected sample audio data, which is not conducive to unified processing of the sample audio data, thereby causing great inconvenience to the processing of the sample audio data. In order to solve the above technical problems, a generalized audio feature extraction method is used to extract features from the sample audio data. For example, an ASR (Automatic Speech Recognition) model is used to extract features from the sample audio data to obtain sample audio features.
[0099] Optionally, when extracting features from the sample audio data, any one of the models such as masr (Magical Automatic Speech Recognition, automatic speech recognition framework) and deepspeech (Deep Speech) can be used to extract features from the sample audio data.
[0100] Optionally, since there is at least one language in the sample audio data, corresponding to at least one audio receiving device and at least one user, when extracting features from the sample audio data, the features of the middle layer of the masr model are used as sample audio features to improve the generalization of feature extraction and thereby improve the efficiency of feature extraction.
[0101] In step 203, the terminal convolves the sample audio features based on the audio to be trained to drive the three-dimensional facial animation model to obtain initial audio features; and encodes the sample speaking style data based on the audio to be trained to drive the three-dimensional facial animation model to obtain sample speaking style features.
[0102] Before step 203, the method further includes: the terminal inputting the sample audio features, the sample speech style data and the corresponding sample BS values into the audio-driven three-dimensional human face animation model to be trained.
[0103] In some embodiments, Figure 2C As shown, the above step 203 includes the following steps:
[0104] In step 2031, the terminal drives the three-dimensional facial animation model to perform convolution on the sample audio feature based on the audio to be trained to obtain at least one intermediate audio feature.
[0105] In some embodiments, when the terminal drives the three-dimensional face animation model to be trained based on the audio to perform convolution on the sample audio feature, the terminal obtains at least one convolution group, each convolution group in the at least one convolution group shares hyperparameters and includes at least one convolution layer. Each convolution group in the at least one convolution group is provided with a convolution calculation channel, and the convolution calculation channel is used to calculate the channel of the input feature (a part of the sample audio feature).
[0106] In step 2032, the terminal performs interval calculation on the at least one intermediate audio feature to obtain the initial audio feature.
[0107] In some embodiments, each convolution layer in at least one convolution layer includes at least one convolution kernel. Although different convolution kernels correspond to different convolution effects, due to the validity of the convolution parameters, there may be a certain correlation between the intermediate audio features corresponding to different convolution groups in at least one convolution group, which leads to an overfitting effect when calculating the intermediate audio features corresponding to different convolution groups, thereby reducing the accuracy of the calculation. In order to solve the aforementioned problem and improve the accuracy of the calculation, the terminal performs interval calculation on the at least one intermediate audio feature to obtain the initial audio feature.
[0108] In some embodiments, Figure 2D As shown, the above step 2032 includes the following steps:
[0109] In step 20321, the terminal drives the three-dimensional facial animation model to be trained based on the audio to match every two intermediate audio features of the at least one intermediate audio feature, and obtains an intermediate audio feature group corresponding to every two intermediate audio features of the at least one intermediate audio feature; wherein each intermediate audio feature of the at least one intermediate audio feature corresponds to a convolution calculation channel, and the sequence values of the two convolution calculation channels corresponding to the intermediate audio feature group are not adjacent.
[0110] In some embodiments, a convolution group in at least one convolution group is used as an example for explanation. When the convolution group performs convolution on the corresponding input feature, the corresponding convolution calculation channel is calculated with the channel of the corresponding input feature to obtain the convolution result of the input feature. Optionally, a part of the sample audio feature is used as an example for explanation. When the convolution group performs convolution on a part of the sample audio feature, the convolution calculation channel corresponding to the convolution group is calculated with the channel corresponding to the part of the sample audio feature to obtain the intermediate audio feature corresponding to the part of the sample audio feature.
[0111] Optionally, taking two convolution groups in at least one convolution group as an example for explanation, if the sequence values of the convolution calculation channels corresponding to the two convolution groups are adjacent, an overfitting problem will occur when calculating the two intermediate audio features corresponding to the two convolution groups. Optionally, in order to solve the overfitting problem, an embodiment of the present application performs pairwise matching on each intermediate audio feature in at least one intermediate audio feature based on the sequence values of the convolution calculation channels of the convolution groups corresponding to each intermediate audio feature in at least one intermediate audio feature to obtain at least one intermediate audio feature group.
[0112] In some embodiments, an intermediate audio feature group in at least one intermediate audio feature group is taken as an example for explanation. The intermediate audio feature group corresponds to two convolution groups, and the two convolution groups correspond to two convolution calculation channels. The two convolution calculation channels are not adjacent, that is, the difference between the sequence values of the two convolution calculation channels is greater than "1". Then, when calculating the two intermediate audio features in the intermediate audio feature group, overfitting will not occur, thereby solving the problem of overfitting.
[0113] In some embodiments, when every two intermediate audio features in the at least one intermediate audio feature are matched based on a preset first matching rule, the difference between the sequence values of the convolution calculation channels corresponding to every two intermediate audio features in the at least one intermediate audio feature is greater than "1". Optionally, two intermediate audio feature groups in the at least one intermediate audio feature group are used as an example for explanation, and two intermediate audio feature groups, one of which has a sequence value of "1" for the convolution calculation channel of the convolution group corresponding to one intermediate audio feature and a sequence value of "6" for the convolution calculation channel of the convolution group corresponding to the other intermediate audio feature, are matched to obtain an intermediate audio feature group.
[0114] In step 20322, the terminal drives the three-dimensional facial animation model to be trained based on the audio to merge the two intermediate audio features in each intermediate audio feature group in at least one intermediate audio feature group to obtain an intermediate merged feature corresponding to each intermediate audio feature group in at least one intermediate audio feature group.
[0115] Through the analysis of step 20321, it can be seen that each of the at least one intermediate merged feature obtained in step 20322 does not have an overfitting problem, thereby improving the accuracy of the at least one intermediate merged feature.
[0116] In step 20323, the terminal calculates each of the at least one intermediate merged feature based on the audio-driven three-dimensional facial animation model to be trained to obtain the initial audio feature.
[0117] In some embodiments, when training the audio-driven 3D facial animation model to be trained, iterative training is required to improve the accuracy of the trained audio-driven 3D facial animation model. Optionally, in order to make the convolution results of different convolution groups in at least one convolution group more evenly integrated, after completing step 20323, before the next round of training, every two intermediate audio features in the at least one intermediate audio feature can be re-matched based on a preset second matching rule to obtain an intermediate audio feature group corresponding to every two intermediate audio features in the at least one intermediate audio feature.
[0118] Optionally, after completing step 20323, during the next round of training, every two intermediate audio features in the at least one intermediate audio feature can be re-matched based on a preset third matching rule to obtain an intermediate audio feature group corresponding to every two intermediate audio features in the at least one intermediate audio feature.
[0119] Optionally, taking two intermediate audio feature groups in at least one intermediate audio feature group as an example for explanation, two intermediate audio feature groups, in which the sequence value of the convolution calculation channel of the convolution group corresponding to one intermediate audio feature is "1" and the sequence value of the convolution calculation channel of the convolution group corresponding to the other intermediate audio feature is "3", are matched to obtain an intermediate audio feature group.
[0120] In step 20323, since each intermediate merged feature in at least one intermediate merged feature does not have overfitting, the initial audio feature obtained based on each intermediate merged feature in at least one intermediate merged feature does not have overfitting, thereby improving the accuracy of the initial audio feature.
[0121] In some embodiments, the above step 203 includes: the terminal performs one-hot encoding on the sample speech style data based on the audio-driven three-dimensional facial animation model to be trained to obtain the sample speech style features.
[0122] It should be noted that one-hot encoding is a conventional encoding method, and the embodiment of the present application does not elaborate on the process of one-hot encoding.
[0123] In step 204, the terminal drives the three-dimensional facial animation model based on the audio to be trained to encode the initial audio features and the sample speaking style features to obtain an output mixed deformation value.
[0124] In some embodiments, Figure 2E As shown, the above step 204 includes the following steps:
[0125] In step 2041, the terminal drives the three-dimensional facial animation model based on the audio to be trained to superimpose the initial audio features and the sample speaking style features to obtain sample superimposed features.
[0126] In step 2041, the three-dimensional facial animation model is driven based on the audio to be trained, and the initial audio features and the sample speaking style features are superimposed in the channel dimension to ensure that the sample superimposed features include rich audio information and the personalized information of the user corresponding to the sample superimposed features, thereby achieving better reproduction of lip animation and facial expressions.
[0127] In step 2042, the terminal encodes the sample superposition feature based on the audio-driven three-dimensional facial animation model to be trained to obtain the output mixed deformation value.
[0128] In some embodiments, Figure 2F As shown, the above step 2042 includes the following steps:
[0129] In step 20421, the terminal encodes the sample superimposed features based on the audio-driven three-dimensional facial animation model to be trained to obtain sample encoded features.
[0130] In some embodiments, the terminal encodes the sample superimposed features of the audio-driven three-dimensional facial animation model to be trained based on a preset encoding model.
[0131] Optionally, in the process of encoding the sample superposition feature based on the preset coding model, the sample superposition feature is divided into different levels, each level representing a different granularity. For example, the sample superposition feature can be gradually downsampled to obtain representations of different levels.
[0132] Optionally, for a level, the preset encoding model uses an independent self-attention mechanism to capture the internal relationship of the level. The level self-attention calculates a weight matrix based on the position, and then multiplies it with the value matrix within the level to obtain the self-attention representation within the level. The self-attention representations of different levels are connected together so that the preset encoding model can pay attention to information of different granularities at the same time. Optionally, the self-attention representations of different levels can be connected together by splicing or weighted averaging.
[0133] Optionally, different levels may interact with each other, so that the preset coding model can capture features of at least one granularity at the same time to improve the performance of the preset coding model.
[0134] Optionally, the preset encoding model compresses model parameters by patient knowledge distillation to minimize the number of parameters while maintaining performance and learn a more concise representation from the teacher model. Secondly, the feedforward neural network layers are reduced and more self-attention mechanisms are added (for example, hierarchical self-attention mechanisms that process information of different granularities), so that the preset encoding model can better understand the input features. In addition, useless encoder and decoder layers are reduced, so that the size of the preset encoding model is reduced and the inference speed is increased, so that the influence relationship between tags can be better understood.
[0135] Optionally, the preset encoding model can be X-Transformer (Deep Transformer fine-tuned for Extremely Large Multi-label Classification Problem (XMC)).
[0136] It should be noted that the method of patient knowledge distillation and the teacher model are conventional methods or models, and the embodiments of this application will not be repeated here.
[0137] From the above analysis, it can be seen that when the sample superposition feature is encoded based on the improved preset coding model, the sample coding feature with higher accuracy can be obtained more quickly.
[0138] In step 20422, the terminal decodes the sample encoding features based on the audio-driven three-dimensional facial animation model to be trained to obtain an output mixed deformation value.
[0139] In some embodiments, the sample encoded feature may be decoded by a Transformer-based decoder to obtain an output blend deformation value.
[0140] It should be noted that the Transformer decoder decodes the sample coding features based on a conventional decoding algorithm, and the embodiments of the present application will not describe the decoding process of the sample coding features in detail.
[0141] In step 205, the terminal calculates the sample mixed deformation value and the output mixed deformation value to obtain a loss function value.
[0142] In some embodiments, the sample mixed deformation value and the output mixed deformation value are calculated based on a preset loss function to obtain a loss function value. Optionally, the preset loss function may be an L1 loss function (absolute value error loss function).
[0143] Optionally, in order to improve the accuracy of the trained audio-driven 3D facial animation model, errors of order 1 to 3 are calculated, wherein the errors of order 1 to 3 include reconstruction error, velocity error, and acceleration error.
[0144] In step 206, the terminal updates the model parameters of the audio-driven 3D facial animation model to be trained based on the loss function value.
[0145] In some embodiments, in step 201, at least one sample data may also be obtained. After obtaining at least one sample data, the above steps 202 to 206 are repeatedly performed so as to iteratively train the audio-driven three-dimensional facial animation model to be trained through at least one sample data, thereby improving the accuracy of the trained audio-driven three-dimensional facial animation model.
[0146] In some embodiments, after obtaining a trained audio-driven three-dimensional facial animation model, the audio to be processed is obtained through an audio receiving device, feature extraction is performed on the audio to be processed based on a preset model to obtain features of the audio to be processed, preset speaking style data is obtained, the features of the audio to be processed and the preset speaking style data are input into the trained audio-driven three-dimensional facial animation model, and at least one corresponding BS value is output; wherein each frame of the image corresponds to at least one BS value, and the complexity of the image is different, and the number of corresponding BS values is different.
[0147] In some embodiments, after obtaining the corresponding BS value, the BS value is input into a preset Unreal Engine, in which various preset scenes already exist. After receiving the BS value, the preset Unreal Engine renders at least one frame of target image (including high-precision lip shape and facial expression) corresponding to the audio feature to be processed onto a video display device based on the preset various scenes and BS value. The facial expression of the person on the target image is consistent with the preset speaking style data. For example, if the preset speaking style data is "exaggerated", the person on the target image corresponds to an exaggerated facial expression.
[0148] It should be noted that the above embodiments are described by taking the application in a terminal as an example. In addition, the embodiments of the present application can also be applied to a server.
[0149] In an embodiment of the present application, sample data of a user is obtained, and the sample data includes sample audio data, sample speaking style data, and sample blending deformation values corresponding to the sample audio data, wherein the sample speaking style data describes the facial expression data of the user, so as to fuse the personalized data of the user with the sample audio data, and retain the personalized data of the user as much as possible. Based on the sample data, the audio-driven three-dimensional facial animation model to be trained is trained, and the trained audio-driven three-dimensional facial animation model obtained can output a high-precision BS value, thereby improving the accuracy of the audio-driven three-dimensional facial animation technology. In addition, the output high-precision BS value is input into a preset Unreal Engine, so that the preset Unreal Engine can render high-precision lip shape and facial expression on a video display device, thereby achieving high-precision reproduction of lip shape and facial expression at the same time, and then the audio-driven three-dimensional facial animation technology can be widely used in scenes with high expression accuracy requirements such as film and television production.
[0150] Figure 3 3 is a schematic diagram of a structure of a device 300 for generating an audio-driven 3D facial animation model according to an embodiment of the present application, the device comprising:
[0151] An acquisition module 301 is used to acquire sample data, the sample data including sample audio data, sample speech style data and sample mixed deformation value, the sample audio data and the sample speech style data belong to the same user, and the sample mixed deformation value is obtained by preprocessing the sample audio data;
[0152] A feature extraction module 302 is used to extract features from the sample audio data to obtain sample audio features;
[0153] The first training module 303 is used to drive the three-dimensional facial animation model based on the audio to be trained to perform convolution on the sample audio feature to obtain the initial audio feature; and drive the three-dimensional facial animation model based on the audio to be trained to encode the sample speech style data to obtain the sample speech style feature;
[0154] The second training module 304 is used to drive the three-dimensional facial animation model to encode the initial audio features and the sample speech style features based on the audio to be trained to obtain an output mixed deformation value;
[0155] A calculation module 305 is used to calculate the sample mixed deformation value and the output mixed deformation value to obtain a loss function value;
[0156] The updating module 306 is used to update the model parameters of the audio-driven 3D facial animation model to be trained based on the loss function value.
[0157] In a possible implementation, the feature extraction module 302 includes:
[0158] The feature extraction unit is used to extract features from the sample audio data based on a preset model, and use the features of the middle layer of the preset model as the sample audio features.
[0159] In a possible implementation, the first training module 303 includes:
[0160] A convolution unit, configured to drive the three-dimensional face animation model to perform convolution on the sample audio feature based on the audio to be trained, so as to obtain at least one intermediate audio feature;
[0161] The interval calculation unit is used to perform interval calculation on the at least one intermediate audio feature to obtain the initial audio feature.
[0162] In a possible implementation, the interval calculation unit includes:
[0163] A first matching subunit is used to drive the three-dimensional facial animation model to match every two intermediate audio features of the at least one intermediate audio feature based on the audio to be trained, so as to obtain an intermediate audio feature group corresponding to every two intermediate audio features of the at least one intermediate audio feature, wherein channel sequence values of the two intermediate audio features in the intermediate audio feature group are not adjacent;
[0164] a merging subunit, configured to drive the three-dimensional facial animation model to merge two intermediate audio features in each intermediate audio feature group in at least one intermediate audio feature group based on the audio to be trained, so as to obtain an intermediate merged feature corresponding to each intermediate audio feature group in the at least one intermediate audio feature group;
[0165] The calculation subunit is used to drive the three-dimensional facial animation model to calculate each intermediate merged feature in at least one intermediate merged feature based on the audio to be trained to obtain the initial audio feature.
[0166] In a possible implementation, the first training module 303 includes:
[0167] The one-hot encoding module is used to perform one-hot encoding on the sample speaking style data based on the audio-driven three-dimensional facial animation model to be trained to obtain the speaking style features of the sample; the speaking style features of the sample are used to describe the user's speaking style.
[0168] In a possible implementation, the second training module 304 includes:
[0169] A superposition unit, used for driving a three-dimensional facial animation model based on the audio to be trained to superimpose the initial audio feature and the sample speaking style feature to obtain a sample superposition feature;
[0170] The encoding unit is used to encode the sample superposition feature based on the audio-driven three-dimensional human face animation model to be trained to obtain the output mixed deformation value.
[0171] In a possible implementation, the encoding unit includes:
[0172] An encoding subunit, used for encoding the sample superimposed features based on the audio-driven three-dimensional human face animation model to be trained to obtain sample encoding features;
[0173] The decoding subunit is used to decode the sample encoding features based on the audio to be trained to drive the three-dimensional facial animation model to obtain an output mixed deformation value.
[0174] It should be noted that: when the audio-driven 3D facial animation model generation device provided in the above embodiment executes the corresponding steps, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the audio-driven 3D facial animation model generation device provided in the above embodiment and the audio-driven 3D facial animation model generation method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment, which will not be repeated here.
[0175] In an embodiment of the present application, sample data of a user is obtained, and the sample data includes sample audio data, sample speaking style data, and sample blending deformation values corresponding to the sample audio data, wherein the sample speaking style data describes the facial expression data of the user, so as to fuse the personalized data of the user with the sample audio data, and retain the personalized data of the user as much as possible. Based on the sample data, the audio-driven three-dimensional facial animation model to be trained is trained, and the trained audio-driven three-dimensional facial animation model obtained can output a high-precision BS value, thereby improving the accuracy of the audio-driven three-dimensional facial animation technology. In addition, the output high-precision BS value is input into a preset Unreal Engine, so that the preset Unreal Engine can render high-precision lip shape and facial expression on a video display device, thereby achieving high-precision reproduction of lip shape and facial expression at the same time, and then the audio-driven three-dimensional facial animation technology can be widely used in scenes with high expression accuracy requirements such as film and television production.
[0176] An embodiment of the present application also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the method for generating an audio-driven three-dimensional facial animation model as described above is implemented.
[0177] Taking electronic devices as terminals as an example, Figure 4 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present application, see Figure 4 The terminal 400 may be a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer or a desktop computer. The terminal 400 may also be called a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.
[0178] Typically, the terminal 400 includes a processor 401 and a memory 402 .
[0179] The processor 401 may include one or more processing cores, such as a 4-core processor, a 4-core processor, etc. The processor 401 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 401 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 401 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 401 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.
[0180] The memory 402 may include one or more computer-readable storage media, which may be non-transitory. The memory 402 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 402 is used to store at least one program code, which is used to be executed by the processor 401 to implement the process executed by the terminal in the method for generating an audio-driven three-dimensional facial animation model provided in the method embodiment of the present application.
[0181] In some embodiments, the terminal 400 may further optionally include: a peripheral device interface 403 and at least one peripheral device. The processor 401, the memory 402 and the peripheral device interface 403 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 403 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 404, a display screen 405, a camera assembly 406, an audio circuit 407 and a power supply 408.
[0182] The peripheral device interface 403 can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 401 and the memory 402. In some embodiments, the processor 401, the memory 402 and the peripheral device interface 403 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 401, the memory 402 and the peripheral device interface 403 can be implemented on a separate chip or circuit board, which is not limited in the embodiments of the present application.
[0183] The radio frequency circuit 404 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 404 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 404 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. In some embodiments, the radio frequency circuit 404 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 404 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 404 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.
[0184] The display screen 405 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 405 is a touch display screen, the display screen 405 also has the ability to collect touch signals on the surface or above the surface of the display screen 405. The touch signal can be input to the processor 401 as a control signal for processing. At this time, the display screen 405 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 405 can be one, set on the front panel of the terminal 400; in other embodiments, the display screen 405 can be at least two, respectively set on different surfaces of the terminal 400 or in a folding design; in other embodiments, the display screen 405 can be a flexible display screen, set on the curved surface or folding surface of the terminal 400. Even, the display screen 405 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 405 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).
[0185] The camera assembly 406 is used to capture images or videos. In some embodiments, the camera assembly 406 includes a front camera and a rear camera. Typically, the front camera is disposed on the front panel of the terminal, and the rear camera is disposed on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 406 may also include a flash. The flash may be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.
[0186] The audio circuit 407 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 401 for processing, or input them into the radio frequency circuit 404 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 400. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 401 or the radio frequency circuit 404 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 407 may also include a headphone jack.
[0187] The power supply 408 is used to power various components in the terminal 400. The power supply 408 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 408 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.
[0188] In some embodiments, the terminal 400 further includes one or more sensors 409 . The one or more sensors 409 include, but are not limited to: an acceleration sensor 410 , a gyroscope sensor 411 , a pressure sensor 44 , an optical sensor 413 , and a proximity sensor 414 .
[0189] The acceleration sensor 410 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal 400. For example, the acceleration sensor 410 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 401 can control the display screen 405 to display the user page in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 410. The acceleration sensor 410 can also be used for collecting game or user motion data.
[0190] The gyro sensor 411 can detect the body direction and rotation angle of the terminal 400, and the gyro sensor 411 can cooperate with the acceleration sensor 410 to collect the user's three-dimensional actions on the terminal 400. The processor 401 can implement the following functions based on the data collected by the gyro sensor 411: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.
[0191] The pressure sensor 412 can be set on the side frame of the terminal 400 and / or the lower layer of the display screen 405. When the pressure sensor 412 is set on the side frame of the terminal 400, it can detect the user's holding signal of the terminal 400, and the processor 401 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 412. When the pressure sensor 412 is set on the lower layer of the display screen 405, the processor 401 controls the operability controls on the UI page according to the user's pressure operation on the display screen 405. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.
[0192] The optical sensor 413 is used to collect the ambient light intensity. In one embodiment, the processor 401 can control the display brightness of the display screen 405 according to the ambient light intensity collected by the optical sensor 413. Specifically, when the ambient light intensity is high, the display brightness of the display screen 405 is increased; when the ambient light intensity is low, the display brightness of the display screen 405 is reduced. In another embodiment, the processor 401 can also dynamically adjust the shooting parameters of the camera assembly 406 according to the ambient light intensity collected by the optical sensor 413.
[0193] The proximity sensor 414, also called a distance sensor, is usually arranged on the front panel of the terminal 400. The proximity sensor 414 is used to collect the distance between the user and the front of the terminal 400. In one embodiment, when the proximity sensor 414 detects that the distance between the user and the front of the terminal 400 is gradually decreasing, the processor 401 controls the display screen 405 to switch from the screen-on state to the screen-off state; when the proximity sensor 414 detects that the distance between the user and the front of the terminal 400 is gradually increasing, the processor 401 controls the display screen 405 to switch from the screen-off state to the screen-on state.
[0194] Those skilled in the art will understand that Figure 4 The structure shown in the figure does not constitute a limitation on the terminal 400, and the terminal 400 may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.
[0195] Take the electronic device as a server as an example. Figure 5It is a structural diagram of a server provided in an embodiment of the present application. The server 500 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 501 and one or more memories 502, wherein the one or more memories 502 store at least one computer program, and the at least one computer program is loaded and executed by the one or more processors 501 to implement the above-mentioned audio-driven three-dimensional human face animation model generation method. Of course, the server 500 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server 500 may also include other components for realizing device functions, which will not be repeated here.
[0196] The embodiment of the present application also provides a computer-readable storage medium, the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned method for generating an image processing model. Optionally, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.
[0197] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.
[0198] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.
Claims
1. A method for generating an audio-driven 3D facial animation model. It is characterized in that include: Acquire sample data, the sample data including sample audio data, sample speaking style data and sample mixed deformation value, the sample audio data and the sample speaking style data belong to the same user, the sample speaking style data is used to describe the user's facial expression, and the sample mixed deformation value is obtained by preprocessing the sample audio data; Extracting features from the sample audio data to obtain sample audio features; Convolving the sample audio features based on the audio to be trained to drive the three-dimensional facial animation model to obtain at least one intermediate audio feature; Based on the audio to be trained, driving the three-dimensional facial animation model, matching every two intermediate audio features of the at least one intermediate audio feature, to obtain an intermediate audio feature group corresponding to every two intermediate audio features of the at least one intermediate audio feature; Based on the audio-driven three-dimensional facial animation model to be trained, two intermediate audio features in each intermediate audio feature group in at least one intermediate audio feature group are merged to obtain intermediate merged features corresponding to each intermediate audio feature group in at least one intermediate audio feature group; Based on the audio to be trained, the three-dimensional facial animation model is driven to calculate each intermediate merged feature in at least one intermediate merged feature to obtain an initial audio feature; each intermediate audio feature in the at least one intermediate audio feature corresponds to a convolution calculation channel, and sequence values of two convolution calculation channels corresponding to the intermediate audio feature group are not adjacent; and encoding the sample speech style data based on the audio-driven three-dimensional human face animation model to be trained to obtain sample speech style features; Encoding the initial audio features and the sample speaking style features based on the audio-driven three-dimensional facial animation model to be trained to obtain an output mixed deformation value; Calculating the sample mixed deformation value and the output mixed deformation value to obtain a loss function value; Model parameters of the audio-driven three-dimensional facial animation model to be trained are updated based on the loss function value.
2. The method according to claim 1, It is characterized in that The extracting features from the sample audio data to obtain sample audio features includes: Feature extraction is performed on the sample audio data based on a preset model, and features of an intermediate layer of the preset model are used as the sample audio features.
3. The method according to claim 1, It is characterized in that The step of encoding the sample speech style data based on the audio-driven three-dimensional human face animation model to be trained to obtain the sample speech style features includes: The sample speech style data is one-hot encoded based on the audio-driven three-dimensional facial animation model to be trained to obtain the sample speech style features.
4. The method according to claim 1, It is characterized in that The step of encoding the initial audio features and the sample speech style features based on the audio to be trained and driving the three-dimensional facial animation model to obtain an output mixed deformation value includes: Superimposing the initial audio features and the sample speaking style features based on the audio-driven three-dimensional facial animation model to be trained to obtain sample superimposed features; The sample superposition feature is encoded based on the audio-driven three-dimensional facial animation model to be trained to obtain the output blend deformation value.
5. The method according to claim 4, It is characterized in that The step of encoding the sample superposition features based on the audio-driven three-dimensional human face animation model to be trained to obtain the output blend deformation value includes: Encoding the sample superposition features based on the audio-driven three-dimensional facial animation model to be trained to obtain sample encoding features; The sample encoding features are decoded based on the audio-driven three-dimensional facial animation model to be trained to obtain an output blending deformation value.
6. A device for generating an audio-driven three-dimensional facial animation model, It is characterized in that include: An acquisition module, used to acquire sample data, wherein the sample data includes sample audio data, sample speaking style data and sample mixed deformation values, wherein the sample audio data and the sample speaking style data belong to the same user, the sample speaking style data is used to describe the user's facial expression, and the sample mixed deformation value is obtained by preprocessing the sample audio data; A feature extraction module, used to extract features from the sample audio data to obtain sample audio features; A first training module is used to drive the three-dimensional facial animation model to perform convolution on the sample audio feature based on the audio to be trained to obtain at least one intermediate audio feature; Based on the audio to be trained, driving the three-dimensional facial animation model, matching every two intermediate audio features of the at least one intermediate audio feature, to obtain an intermediate audio feature group corresponding to every two intermediate audio features of the at least one intermediate audio feature; Based on the audio-driven three-dimensional facial animation model to be trained, two intermediate audio features in each intermediate audio feature group in at least one intermediate audio feature group are merged to obtain intermediate merged features corresponding to each intermediate audio feature group in at least one intermediate audio feature group; Based on the audio to be trained, the three-dimensional facial animation model is driven to calculate each intermediate merged feature in at least one intermediate merged feature to obtain an initial audio feature; each intermediate audio feature in the at least one intermediate audio feature corresponds to a convolution calculation channel, and sequence values of two convolution calculation channels corresponding to the intermediate audio feature group are not adjacent; and encoding the sample speech style data based on the audio-driven three-dimensional human face animation model to be trained to obtain sample speech style features; A second training module is used to drive the three-dimensional facial animation model to encode the initial audio features and the sample speaking style features based on the audio to be trained to obtain an output mixed deformation value; A calculation module, used to calculate the sample mixed deformation value and the output mixed deformation value to obtain a loss function value; An updating module is used to update the model parameters of the audio-driven three-dimensional facial animation model to be trained based on the loss function value.
7. An electronic device, It is characterized in that The electronic device includes a processor and a memory, the memory is used to store at least one program, and the at least one program is loaded by the processor and executed according to the method for generating an audio-driven three-dimensional facial animation model as described in any one of claims 1 to 5.
8. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores at least one program, and the at least one program is loaded and executed by the processor to implement the method for generating an audio-driven three-dimensional facial animation model according to any one of claims 1 to 5.
Citation Information
Patent Citations
Multi-channel neural network model training method, device and computer storage medium
CN111667045A
Face driving model training and face mouth shape animation generation method
CN112396182A
GTCN-based real-time voice emotion recognition method and application device
CN113257279A
Mixed deformation value output method and device, storage medium and electronic device
CN113592985A
Audio-driven expression method and device based on causal convolution
CN115272537A
