A method for transferring human facial expressions to virtual character faces

Abstract

The present invention discloses a method for migrating facial expressions to the face of a virtual character. The method requires a user to provide a small number of grid pairs that match human expressions with virtual character expressions, and the grid pairs, together with a large number of grids with random expressions that are automatically generated, are used as training data for a neural network. The neural network learns to obtain a mapping that matches the facial expressions of any human grid with those of the virtual character grid. The present invention significantly reduces the workload of the expression migration workflow, and the number of expression matching grid pairs that the user needs to create is no longer as many as dozens of pairs. After the neural network training is completed, the method of the present invention can migrate the expression of any human face grid to the grid of the virtual character in real time, and the migration result can ensure that it is consistent with the human face expression in terms of sensory perception.

Description

Technical Field

[0001] The invention relates to the field of three-dimensional geometric modeling, and in particular to a method for migrating human facial expressions to a virtual character's face. Background Art

[0002] Expression transfer technology has been widely used in the fields of live broadcasting, film industry, social software and VR applications. Previous work used blendshape deformation animation technology to achieve expression transfer. This method requires artists to manually create a large number of virtual character expression meshes corresponding to basic facial expressions, which has high requirements for manual workload, professional skills and experience. The method based on geometric features calculates the deformation features of facial expressions relative to the expressionless mesh, and applies the same deformation to the expressionless mesh of the virtual character to transfer expressions. However, this method is only applicable to meshes with similar appearances, while the facial features of virtual characters are often very different from those of human faces. For example, a cartoon-style virtual character has large eyes, and the virtual character cannot completely close its eyes in the transfer results of this method. The recent expression transfer method is based on machine learning technology. The unsupervised method does not require manual labor, but uses geometric features as a priori, so it has similar problems as the method based on geometric features; the supervised method uses a large amount of manual annotation as a priori. Although it reduces the requirements for professionalism, it still requires a certain amount of labor and it is difficult to obtain high-quality expression results that meet the requirements of industrial production. Summary of the invention

[0003] The purpose of the present invention is to provide a method for migrating facial expressions to virtual character faces in view of the deficiencies of the prior art. The present invention uses a deep learning method to calculate the position and Laplace operator of each expression grid in a low-dimensional space, and aligns the expression manifolds of the human face and the virtual character in the low-dimensional space according to a small number of expression matching grid pairs provided by the user, thereby finding a mapping of facial expression matching from the human face to the virtual character. The present invention can significantly reduce the manual workload required for expression migration and has high practical value.

[0004] The objective of the present invention is achieved through the following technical solutions:

[0005] A method for transferring human facial expressions to a virtual character's face comprises the following steps:

[0006] (1) preparing a training data set: the training data set includes face mesh data, virtual character mesh data, and an expression matching mesh pair provided by a user; wherein the face mesh data is obtained by randomly sampling expression coefficients of a public face data set; the virtual character mesh data is obtained by randomly applying reasonable deformation processing to the virtual character; the expression matching mesh pair includes a face mesh with expression and a virtual character mesh with the same expression as the face;

[0007] (2) respectively training a variational autoencoder network using the face mesh data and the virtual character mesh data, encoding each mesh using the trained variational autoencoder network to obtain a low-dimensional implicit vector corresponding to each mesh, and finding the K nearest neighbor meshes of each mesh using the Euclidean distance of the low-dimensional implicit vector as a criterion;

[0008] (3) using the training data set to jointly train a variational autoencoder network corresponding to a new human face and a variational autoencoder network corresponding to a new virtual character, during the training process, embedding the training data set into a low-dimensional space, and aligning the manifold of the human face mesh data in the low-dimensional space with the manifold of the virtual character according to the position of the expression matching mesh pair in the low-dimensional space and the Laplace operator; the Laplace operator is the average value of the features of the mesh itself minus the features of K neighboring meshes;

[0009] (4) using the training data set to train a cyclic generative adversarial network to learn a mapping between a human face mesh and a facial expression of a virtual character mesh;

[0010] (5) The facial mesh of the expression to be transferred is input into the encoder of the variational autoencoder network corresponding to the trained face to obtain its corresponding low-dimensional implicit vector, and then the low-dimensional implicit vector is input into the trained recurrent generative adversarial network to obtain the low-dimensional implicit vector of the virtual character, and then the low-dimensional implicit vector of the virtual character is input into the decoder of the variational autoencoder network corresponding to the trained virtual character to obtain the virtual character with the transferred expression.

[0011] Furthermore, in the step (3), during the manifold alignment process of the face mesh data and the virtual character, the position of the face mesh data in the constrained expression matching mesh pair corresponds to the position of the virtual character mesh, and the Laplacian operator of the face mesh data in the expression matching mesh pair corresponds to the Laplacian operator of the virtual character mesh.

[0012] Furthermore, in the step (4), when the training cycle generates the adversarial network, a human face mesh and a virtual character mesh are randomly selected from the training data set, and the selected human face mesh and virtual character mesh, as well as their respective K nearest neighbor meshes, are combined into the training data for the batch.

[0013] Furthermore, in the step (4), when training the cyclic generative adversarial network, the parameters of the two variational autoencoder networks trained in step (3) are fixed, and the outputs of the encoders of the two variational autoencoder networks after training are used as the input of the cyclic generative adversarial network, and at the same time, the position and Laplace operator of the face mesh of the input cyclic generative adversarial network and its corresponding migrated virtual character mesh are constrained to correspond one-to-one, and the position and Laplace operator of the virtual character mesh of the input cyclic generative adversarial network and its corresponding migrated face mesh are constrained to correspond one-to-one.

[0014] The beneficial effects of the present invention are:

[0015] The present invention combines deep learning technology, and the trained neural network can achieve a real-time effect of 30 frames per second in practical applications, so that the present invention can be widely used in scenes with high real-time requirements. In the final result, even if the appearance features of the human face are very different from the face of the virtual character, the present invention can make the expression of the migrated virtual character mesh consistent with the input human face mesh expression in human senses by combining the similarity of the overall expression manifold and the local features of the expression matching grid pair provided by the user, thereby overcoming the problem that the traditional method based on geometric features cannot close the eyes of the large-eyed virtual character. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 : is the algorithm architecture diagram of the present invention; the left figure is a schematic diagram of the expression embedding training process of the perception Laplacian operator; the right figure is a schematic diagram of the expression migration training process of the perception Laplacian operator;

[0017] Figure 2 It is a result diagram of expression migration based on real data of the present invention; from left to right: a real face image, a face expression mesh generated by combining the existing expression tracker, and a face mesh of the virtual character Mery obtained after network migration of the present invention. DETAILED DESCRIPTION

[0018] The present invention will be described in detail below based on the accompanying drawings and preferred embodiments, and the purpose and effects of the present invention will become more clear. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0019] The core technology of the present invention is to use a Laplace-aware variational autoencoder network and a recurrent adversarial generation network to match the expression grid pairs initially provided by the user, and find the corresponding expression mapping from the human face to the virtual character's face that conforms to human senses.

[0020] The present invention comprises the following steps:

[0021] 1. Prepare training data sets: The data required for training neural networks include a large amount of face mesh data with random expressions, a large amount of virtual character mesh data with random expressions, and expression matching mesh pairs provided by users.

[0022] 1.1. Prepare a large amount of face mesh data with random expressions

[0023] Based on the public face dataset, random sampling of expression coefficients can automatically generate a large number of face mesh data with random expressions. The random sampling process of expression coefficients can directly use a random number generator, or use a public face image dataset and combine it with an existing expression tracker to capture the expression coefficients in these images for sampling.

[0024] 1.2. Prepare a large amount of mesh data of virtual characters with random expressions

[0025] Using 3D modeling tools such as Autodesk Maya, multiple sets of 3D deformations can be created for virtual character meshes to simulate the expressions of virtual characters. By randomly sampling the weight coefficients of these 3D deformations within a reasonable range, a large number of virtual character meshes with random expressions can be automatically generated.

[0026] 1.3. Prepare the expression matching grid pairs provided by the user

[0027] Users can use 3D modeling tool software such as Autodesk Maya to manually create 9 pairs of facial meshes with expressions and virtual character meshes with the same expressions as the faces.

[0028] 2. Find the K nearest neighbor grids of each grid in the data set, which is divided into the following steps:

[0029] 2.1. Using a variational autoencoder network to embed the face mesh and the avatar mesh into a low-dimensional space

[0030] In order to facilitate the neural network to learn the deformation of the mesh, each mesh in the training data set described in step 1 is first converted into a DR feature (L.Gao, Y.-K.Lai, J.Yang, L.-X.Zhang, L.Kobbelt, and S.Xia.Sparse datadriven mesh deformation.arXiv preprint arXiv:1709.01250, 2017.) representation. Two variational autoencoder networks are used to learn the encoding of the face mesh and the virtual character mesh in the training data set, and the low-dimensional implicit vector obtained after encoding each mesh is recorded:

[0031] l A =Enc(M A )

[0032] Among them, M A represents the DR feature of the grid A, Enc(·) represents the encoder part of the variational autoencoder network, and l A Represents the low-dimensional implicit vector obtained after encoding the grid.

[0033] 2.2. Finding the K nearest neighbors of a grid

[0034] After obtaining the low-dimensional implicit vector expression of the grid in step 2.1, the KNN (K Nearest Neighbour) algorithm is used to find the K nearest neighbor grids of each grid using the Euclidean distance between implicit vectors as the measurement standard, and the set of the K nearest neighbor grids of grid A is recorded as N(A).

[0035] 3. Expression embedding process of perceptual Laplacian operator: The training data set is used to jointly train a variational autoencoder network corresponding to a new face mesh and a variational autoencoder network corresponding to a new virtual character, so that the manifold of the face mesh data in the low-dimensional space is aligned with the manifold of the virtual character.

[0036] 3.1. Embedding the expression mesh into a low-dimensional manifold

[0037] Just like training the default variational autoencoder network in step 2.1, train two new variational autoencoder networks using mini-batch gradient descent to learn the training dataset:

[0038] l A =Enc(M A ), M′ A = Dec(l A )

[0039] Among them, M A 、Enc(·),l A The meaning of can be found in step 2.1, Dec(·) represents the decoder part of the variational autoencoder network, M′ A Denotes the decoder of the variational autoencoder according to l A The DR features are restored. The loss terms imposed on the network training in this step are:

[0040] L rec =||M′ A -M A ||1

[0041]

[0042] Where L rec represents the reconstruction loss, L kld represents KL divergence loss, KL(·) represents KL divergence calculation function, represents the standard normal distribution, Q(l A |A) indicates l A About the posterior distribution of A.

[0043] 3.2. According to the expression matching mesh pair, the manifold of the face mesh data in the low-dimensional space is aligned with the manifold of the virtual character

[0044] In this step, when training the network at each step, a pair of expression matching grids (A) is randomly selected from the expression matching grid pairs. i , B i ), find their K nearest neighbor grids to calculate the Laplace operator, where A i is the face mesh in the expression matching mesh pair, B i is a virtual character mesh. This pair of meshes and their K nearest neighbors are used as small batch training data, and the variational autoencoder network is allowed to learn them while constraining the network to make A i The position and Laplacian in the low-dimensional manifold are respectively related to B i The position in the low-dimensional manifold is consistent with the Laplace operator, that is, the loss term imposed on the network training is divided by L rec , L kld In addition, it also includes:

[0045]

[0046]

[0047] Where L anchor represents the position constraint of the expression matching mesh pair in the low-dimensional manifold, L anchor-Laplacian represents the Laplace constraint on the expression matching grid pair in the low-dimensional manifold, represents the Laplace operator, which is calculated as:

[0048]

[0049] 3.3. Alternating training of Laplacian-aware variational autoencoder networks

[0050] Referring to steps 2.1 and 2.2, two new variational autoencoder networks are trained alternately.

[0051] 4. Expression transfer process of perceptual Laplacian operator: According to the two low-dimensional manifolds encoded in step 3, the training dataset is used to train a cyclic generative adversarial network to learn the mapping between a human face and the facial expression of the virtual character mesh.

[0052] 4.1. Training Laplacian-aware Recurrent Generative Adversarial Networks

[0053] Since the manifolds of the human face and the virtual character's facial mesh have been roughly aligned in step 3, the expression matching meshes have similar positions and Laplacian operators on the two manifolds, thereby reducing the difficulty of finding the expression matching mapping. The parameters of the two variational autoencoder networks trained in step 3 are fixed, and the output of the encoders of the two trained variational autoencoder networks is used as the input of the recurrent generative adversarial network, which finds the mapping of expression matching between the human face mesh and the virtual character mesh, thereby completing the expression migration work.

[0054] When training the network, taking the migration from human face to virtual character as an example, a pair of grids (A, B) and their K nearest neighbors are randomly selected each time as the small batch training data for this time. The loss term of the constrained cyclic generative adversarial network during training is:

[0055]

[0056]

[0057] L cycle =||l A -T(S(l A ))||2+||l B -S(T(l B ))||2

[0058] L corr =||l A -S(l A )||2+||l B -T(l B )||2

[0059]

[0060] Among them, S is the generator network that migrates from the face mesh to the virtual character mesh, T is the generator network that migrates from the virtual character mesh to the face mesh, Dis human With Dis avatar They are the discriminator networks for judging the authenticity of face meshes and virtual character meshes, respectively. and To combat the loss term, L cycle is the cycle consistency constraint, L corr is the position constraint of the mesh in the manifold before and after migration, L Laplacian Laplace constraints on the mesh in the manifold before and after migration. right There are similar definitions.

[0061] 4.2 Using the trained network for expression transfer

[0062] According to the variational autoencoder network and recurrent generative adversarial network of the perceptual Laplacian operator trained in step 3 and step 4.1, expression migration can be performed: for any face mesh A, the network outputs the DR features of the virtual character mesh after migration

[0063]

[0064] according to Reconstruct the virtual character mesh to obtain virtual character mesh B with the same expression as A. output As a result of the expression migration work.

[0065] Implementation Examples

[0066] The inventors implemented the implementation example of the present invention on a desktop computer equipped with an AMD R7-3800X CPU, an Nvidia GeForce GTX 1660SUPER graphics card, and 16GB of memory. The inventors used the public datasets Facewarehouse and FaceForensics++ to obtain face meshes with random expressions, and used virtual characters publicly released on the Internet. To generate a virtual character mesh with random expressions, and to manually create 9 pairs of expression matching mesh pairs using AutodeskMaya to test the experimental results of the invention. The results show that on the current hardware configuration, the present invention can transfer human facial expressions to the face of the virtual character in real time at a speed of 30 frames per second. In the final transfer result, the expression of the virtual character is consistent with the input human facial expression in the human senses, and the difficulties encountered by the traditional method (for example, the virtual character with big eyes cannot close his eyes) are also solved in this method. Compared with the traditional expression transfer workflow, the user needs to manually create at least 40 pairs of human faces and virtual character mesh pairs with the same expression, and it often takes more than 3 hours to transfer the expression for each virtual character. The present invention utilizes the Laplace operator information of the human face and virtual character expression in the low-dimensional space. The user only needs to manually create 9 pairs of human faces and virtual character mesh pairs with the same expression, which takes only 30 minutes to 60 minutes, significantly reducing the workload of the expression transfer work.

[0067] Those skilled in the art can understand that the above are only preferred examples of the invention and are not intended to limit the invention. Although the invention is described in detail with reference to the above examples, those skilled in the art can still modify the technical solutions recorded in the above examples or replace some of the technical features therein with equivalents. Any modification, equivalent replacement, etc. made within the spirit and principle of the invention shall be included in the protection scope of the invention.

Claims

1. A method for transferring human facial expressions to a virtual character's face, characterized in that: The following steps are involved: (1) Preparing a training data set: The training data set includes face mesh data, virtual character mesh data, and expression matching mesh pairs provided by users; wherein the face mesh data is obtained by randomly sampling expression coefficients of a public face dataset; the virtual character mesh data is obtained by randomly applying reasonable deformation processing to the virtual character; the expression matching mesh pairs include a face mesh with expression and a virtual character mesh with the same expression as the face; (2) respectively training a variational autoencoder network using the face mesh data and the virtual character mesh data, encoding each mesh using the trained variational autoencoder network to obtain a low-dimensional implicit vector corresponding to each mesh, and finding the K nearest neighbor meshes of each mesh using the Euclidean distance of the low-dimensional implicit vector as a measurement standard; (3) using the training data set to jointly train a variational autoencoder network corresponding to a new human face and a variational autoencoder network corresponding to a new virtual character, during the training process, embedding the training data set into a low-dimensional space, and aligning the manifold of the human face mesh data in the low-dimensional space with the manifold of the virtual character according to the position and Laplace operator of the expression matching mesh pair in the low-dimensional space; the Laplace operator is the average value of the features of the mesh itself minus the features of K neighboring meshes; (4) using the training data set to train a cyclic generative adversarial network to learn the mapping between the facial expression of the face mesh and the facial expression of the virtual character mesh; when training the cyclic generative adversarial network, fixing the parameters of the two variational autoencoder networks trained in step (3), using the outputs of the encoders of the two variational autoencoder networks trained as the input of the cyclic generative adversarial network, and constraining the position and Laplacian operator of the face mesh input to the cyclic generative adversarial network to correspond one-to-one with the position and Laplacian operator of the corresponding virtual character mesh after migration, and constraining the position and Laplacian operator of the virtual character mesh input to the cyclic generative adversarial network to correspond one-to-one with the position and Laplacian operator of the corresponding face mesh after migration; (5) The facial mesh of the expression to be transferred is input into the encoder of the variational autoencoder network corresponding to the trained face to obtain its corresponding low-dimensional implicit vector, and then the low-dimensional implicit vector is input into the trained recurrent generative adversarial network to obtain the low-dimensional implicit vector of the virtual character, and then the low-dimensional implicit vector of the virtual character is input into the decoder of the variational autoencoder network corresponding to the trained virtual character to obtain the virtual character with the transferred expression.

2. The method for transferring human facial expressions to a virtual character's face according to claim 1, characterized in that: In the step (3), during the manifold alignment process of the face mesh data and the virtual character, the position of the face mesh data in the constrained expression matching mesh pair corresponds to the position of the virtual character mesh, and the Laplacian operator of the face mesh data in the expression matching mesh pair corresponds to the Laplacian operator of the virtual character mesh.

3. The method for transferring human facial expressions to a virtual character's face according to claim 1, characterized in that: In the step (4), when training the generative adversarial network in a loop, a human face mesh and a virtual character mesh are randomly selected from the training data set, and the selected human face mesh and virtual character mesh, as well as their respective K nearest neighbor meshes, form the training data for the batch.

Images