Animation generation method for tracking facial expressions and neural network training method thereof

Abstract

The present application provides an animation generation method for tracking facial expression and a neural network training method thereof. The animation generation method for tracking facial expression includes: driving a first character model according to a set of expression parameters to obtain a virtual expression image corresponding to the set of expression parameters; applying a plurality of real human facial images to the virtual expression image corresponding to the facial expression to generate a plurality of real human expression images; training a tracking neural network according to the set of expression parameters and the real human expression images; inputting a target facial image into the trained tracking neural network to obtain a predicted set of expression parameters; and controlling a second character model with the predicted set of expression parameters.

Description

Technical Field

[0001] The present invention relates to an animation generation method, and more particularly to an animation generation method for tracking facial expressions and a neural network training method thereof. Background Art

[0002] Currently, creating animations that mimic the user's facial expressions requires animators to adjust the character model's expression control parameters based on the user's facial image and then apply the parameters to the character model. This method is not only time-consuming and laborious, but also difficult to achieve the desired expression, resulting in animations that lack the naturalness and vividness of the user's facial expressions. Summary of the Invention

[0003] In view of this, one embodiment of the present invention proposes an animation generation method for tracking facial expressions, including: driving a first character model according to an expression parameter set to obtain a virtual expression image corresponding to the expression parameter set, wherein the virtual expression image presents a facial expression of a virtual character; applying multiple real-life facial images to the virtual expression images corresponding to the facial expressions to generate multiple real-life expression images; training a tracking neural network based on the expression parameter set and the real-life expression images; inputting a target facial image into the trained tracking neural network to obtain a predicted expression parameter set; and controlling a second character model with the predicted expression parameter set.

[0004] One embodiment of the present invention further provides a neural network training method, comprising: driving a character model according to an expression parameter set to obtain a virtual expression image corresponding to the expression parameter set, wherein the virtual expression image presents a facial expression of a virtual character; applying multiple real-life facial images to the virtual expression image corresponding to the facial expression to generate multiple real-life expression images; and training a neural network based on the expression parameter set and the real-life expression images.

[0005] One embodiment of the present invention further proposes a method for generating animation by tracking facial expressions, comprising: driving a first character model based on multiple expression parameter sets to obtain multiple first virtual expression images corresponding to the expression parameter sets, each of the first virtual expression images presenting a different facial expression of a virtual character; training a tracking neural network based on the expression parameter sets and the first virtual expression images; applying a virtual facial image to a target facial image to generate a second virtual expression image; inputting the second virtual expression image into the trained tracking neural network to obtain a predicted expression parameter set; and controlling a second character model with the predicted expression parameter set.

[0006] According to the neural network training method of the embodiment of the present application, a large number of training samples can be conveniently made. According to the animation generation method for tracking facial expressions of the embodiment of the present application, the corresponding expression parameter set can be directly predicted through the facial image, and the expression of the second role model can be made rich and natural, and then each parameter in the parameter set does not need to be set and repeatedly adjusted.

[0007] The present application is described in detail below with reference to the accompanying drawings and specific embodiments, but is not limited to the present application. BRIEF DESCRIPTION OF DRAWINGS

[0008] Figure 1 The flow chart of the animation generation method for tracking facial expressions of the first embodiment of the present application.

[0009] Figure 2 The software architecture schematic diagram of the first embodiment of the present application.

[0010] Figure 3 The flow chart of the animation generation method for tracking facial expressions of the second embodiment of the present application.

[0011] Figure 4 The partial software architecture schematic diagram of the second embodiment of the present application.

[0012] Figure 5 The flow chart of the animation generation method for tracking facial expressions of the third embodiment of the present application.

[0013] Figure 6 The partial software architecture schematic diagram of the third embodiment of the present application.

[0014] Among them, the reference signs are:

[0015] 20: expression parameter set

[0016] 21: first role model

[0017] 22: virtual expression image

[0018] 23: real facial image

[0019] 24: face changing neural network

[0020] 25: real expression image

[0021] 26: tracking neural network

[0022] 27: target facial image

[0023] 28: predicted expression parameter set

[0024] 29: second role model

[0025] 41: specified facial image

[0026] 42:Specify emoticon image

[0027] 43: First virtual expression image

[0028] 44: Virtual facial image

[0029] 45: Second virtual expression image

[0030] S10, S30, S50, S60, S70, S90: Steps

[0031] S11, S31, S51, S71, S91: Steps DETAILED DESCRIPTION

[0032] The structural principle and working principle of the present invention are described in detail below with reference to the accompanying drawings:

[0033] Some embodiments of the present invention provide a method for generating an animation for tracking facial expressions, a method for training a neural network, a computer device, and a computer-readable storage medium. The computer device includes one or more processors and one or more non-volatile memories. The one or more non-volatile memories store at least one instruction. The at least one instruction is loaded and executed by the one or more processors to implement the method for generating an animation for tracking facial expressions or the method for training a neural network. The processor may be, for example, a processing unit such as a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processing (DSP) chip, a microcontroller (MCU), or a field programmable logic device (FPGA). The computer-readable storage medium is a non-transitory storage medium, such as a flash memory, a hard disk, an optical disk, etc., to store the at least one instruction.

[0034] In some embodiments, the computer device is composed of a single computing device. In other embodiments, the computer device is composed of multiple computing devices that communicate with each other via a network. The computing device can be a mobile computing device (such as a mobile phone or tablet), a computer (such as a personal computer or laptop), a server, etc. It is understood that although not specifically mentioned above, the computer device may also include other hardware components (if necessary) to provide other functions, such as a network module, audio circuit, display screen, camera device, power supply, etc.

[0035] Merge References Figure 1 and Figure 2 . Figure 1 FIG. 1 is a flowchart of a method for generating animation by tracking facial expressions according to a first embodiment of the present invention. Figure 2The software architecture of the first embodiment of the present invention is schematically illustrated. First, a first character model 21 is driven by a game engine (or 3D engine) based on an expression parameter set 20, and a virtual expression image 22 corresponding to the expression parameter set 20 is obtained through rendering (step S10). The first character model 21 is a three-dimensional model. The expression parameter set 20 may be, for example, but not limited to, blendshapes, facial landmarks, action units, or mesh principal component analysis (PCA). The virtual expression image 22 represents a facial expression (e.g., opening the mouth, blinking, etc.) of a virtual character.

[0036] Next, the multiple real-life facial images 23 are applied to the virtual facial images 22 corresponding to the facial expressions, thereby generating multiple real-life facial images 25 (step S30). The real-life facial images 25 have the same facial expressions as the virtual facial images 22. The application (or mapping) refers to giving two parameter spaces (hereinafter referred to as the first space and the second space, which are two-dimensional spaces) corresponding to the real-life facial images 23 and the virtual facial images 22, respectively, so that each point in the first space can find a corresponding point in the second space. Specifically, the application can be implemented using a face-swapping neural network 24, such as FSNet, FSGAN, FaceShifter, FaceController, etc. The face-swapping neural network 24 includes an autoencoder and a generator. The autoencoder includes an encoder and a decoder. The encoder is used to encode and compress the input image to obtain a hidden layer. The decoder then decodes the hidden layer to restore the input image. Here, the autoencoder includes: a first autoencoder for receiving the real-life facial image 23, and a second autoencoder for receiving the virtual facial image 22. The generator is used to synthesize an image by utilizing the hidden layers of the first autoencoder and the second autoencoder to generate a real-life expression image 25 having the same facial expression as the virtual expression image 22 .

[0037] Through steps S10 and S30, real-person facial expression images 25 corresponding to the facial expression parameter set 20 can be obtained. In particular, in step S30, real-person facial images 23 corresponding to different real people are applied to obtain a large number of real-person facial expression images 25 of different faces with the same expression. This facilitates the production of a large number of samples for training the tracking neural network 26.

[0038] In some embodiments, step S10 and step S30 are repeatedly executed, each execution using a different expression parameter set 20 to obtain corresponding real-person expression images 25 of different facial expressions.

[0039] In step S50, a neural network (hereinafter referred to as a tracking neural network 26) is trained according to the expression parameter sets 20 and the real human expression images 25. The tracking neural network 26 can be a convolutional neural network (CNN). The real human expression images 25 are input to the tracking neural network 26, and the expression parameter sets 20 are used as labels of the real human expression images 25. In this way, the tracking neural network 26 trained by step S50 can predict a corresponding expression parameter set (hereinafter referred to as a predicted expression parameter set 28) according to a real human expression image 25. The foregoing steps S10 to S50 are a neural network training method of an embodiment of the present application. In some embodiments, the trained neural network can also be used for other applications, not limited to the foregoing predicted expression parameter set 28.

[0040] In some embodiments, before the real human expression images 25 are input to the tracking neural network 26, the real human expression images 25 can be preprocessed, such as cropping, scaling, filtering, enhancement, etc.

[0041] In step S70, a target face image 27 is input to the trained tracking neural network 26 to obtain a predicted expression parameter set 28. The target face image 27 is an image of a tracking object, such as an image of a user captured by a camera.

[0042] In step S90, a second character model 29 (such as a cartoon character, a celebrity) is controlled by the predicted expression parameter set 28, so that the second character model 29 can present an expression like the target face image 27. The second character model 29 is a three-dimensional model.

[0043] By reference Figure 3 and Figure 4 . Figure 3 is a flowchart of an animation generation method for tracking facial expressions of a second embodiment of the present application. Figure 4This is a partial software architecture diagram of the second embodiment of the present invention. The difference from the first embodiment is that, before step S70, step S60 is included: a designated facial image 41 is applied to a designated expression image 42 to obtain a target facial image 27 for executing step S70. Specifically, the application is achieved using the face-swapping neural network 24 of step S10, where the designated facial image 41 and the designated expression image 42 are input into the face-swapping neural network 24. The designated facial image 41 and the designated expression image 42 are images of the same real person. For example, the designated expression image 42 is a current image of the user captured by a camera, and the designated facial image 41 is another image of the user (e.g., a pre-stored photograph or an image of the user previously captured by the camera). Although the output of the face-swapping neural network 24 is still the facial image of the same user, the image converted by the face-swapping neural network 24 is more closely aligned with the training sample type, making the predicted expression parameter set 28 obtained in step S70 more accurate.

[0044] Merge References Figure 5 and Figure 6 . Figure 5 FIG. 1 is a flowchart of a method for generating animation by tracking facial expressions according to a third embodiment of the present invention. Figure 6 This is a partial software architecture diagram of the third embodiment of the present invention. First, as in the first embodiment, a first character model 21 is driven by a game engine (or 3D engine) based on multiple expression parameter sets 20. This is then rendered to produce multiple first virtual expression images 43 corresponding to these expression parameter sets 20 (step S11). Each first virtual expression image 43 represents a different facial expression of the same virtual character.

[0045] Next, a neural network (here, the tracking neural network 26) is trained based on the expression parameter sets 20 and the first virtual expression images 43 (step S31). Here, the structure of the tracking neural network 26 is as described in the first embodiment and will not be repeated here. It should be noted that the input of the tracking neural network 26 of this embodiment is different from step S50 of the first embodiment. In this embodiment, the first virtual expression image 43 is input to the tracking neural network 26, and the expression parameter set 20 is used as a label for the first virtual expression image 43. The aforementioned steps S11 to S31 are the neural network training method of another embodiment of the present invention. Thereby, the tracking neural network 26 trained by step S31 can obtain the corresponding predicted expression parameter set 28 based on the input virtual expression image. In some embodiments, the trained neural network can also be used for other applications, not limited to the aforementioned predicted expression parameter set 28.

[0046] In step S51, a virtual facial image 44 is applied to a target facial image 27 to generate a second virtual expression image 45. Virtual facial image 44 is the facial image of the first character model 21. As previously mentioned, target facial image 27 is an image of a tracking object, such as an image of a user captured by a camera. This application is the same as previously described and can be implemented using the face-changing neural network 24, so this description will not be repeated here. The difference from the previous embodiment is that in this embodiment, virtual facial image 44 is applied to the target facial image 27. In other words, the second virtual expression image 45 generated by step S51 is a facial image of the first character model 21 having the same facial expression as the target facial image 27.

[0047] In step S71 , the second virtual facial expression image 45 is input to the trained tracking neural network 26 to obtain a predicted facial expression parameter set 28 .

[0048] In step S91, the predicted expression parameter set 28 is used to control a second character model 29 (eg, a cartoon character, celebrity) so that the second character model 29 can present the same expression as the target facial image 27. The second character model 29 is a three-dimensional model.

[0049] In the third embodiment, the target facial image 27 is converted into the facial image of the first character model 21. Therefore, regardless of the user of the target facial image 27, the tracking neural network 26 outputs the predicted expression parameter set 28 based on the same facial image of the first character model 21. This allows for stable prediction results.

[0050] In some embodiments, the first character model 21 and the second character model 29 are identical to each other.

[0051] In some embodiments, the first character model 21 and the second character model 29 are different from each other.

[0052] In some embodiments, multiple first character models 21 may be used to obtain multiple virtual expression images 22 showing facial expressions of different virtual characters, and multiple real face images 23 may be applied to the virtual expression images 22 to train the tracking neural network 26.

[0053] In some embodiments, the predicted expression parameter set 28 obtained by the trained tracking neural network 26 can be used to control a plurality of second character models 29 .

[0054] The neural network training method according to the embodiment of the present invention can conveniently produce a large number of training samples. The animation generation method for tracking facial expressions according to the embodiment of the present invention can directly predict the corresponding expression parameter set from the facial image, without being restricted by a preset expression parameter set, allowing the second character model to express rich and natural expressions.

[0055] Of course, the present invention may have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for generating an animation by tracking facial expressions, characterized in that: include: Driving a first character three-dimensional model according to an expression parameter set and performing rendering to obtain a virtual expression two-dimensional image corresponding to the expression parameter set, the virtual expression two-dimensional image presenting a facial expression of a virtual character; Applying a plurality of real-person face two-dimensional images to the virtual expression two-dimensional images corresponding to the facial expressions using a face-swapping neural network to generate a plurality of real-person expression two-dimensional images, wherein the real-person expression two-dimensional images have the same facial expressions as the virtual expression two-dimensional images, the real-person expression two-dimensional images correspond to the expression parameter set, and the real-person expression two-dimensional images have different faces but the same facial expressions; Repeating the above steps, using different sets of expression parameters, to obtain the corresponding two-dimensional images of real-person expressions of different facial expressions; Training a tracking neural network based on each of the expression parameter sets and the two-dimensional images of real human expressions corresponding to each of the expression parameter sets; Inputting a target facial image into the trained tracking neural network to obtain a predicted expression parameter set; and A second character model is controlled using the predicted expression parameter set.

2. The method for generating an animation by tracking facial expressions as claimed in claim 1, wherein: Before inputting the target facial image into the trained tracking neural network, the method further includes: A designated facial image is applied to a designated expression image to obtain the target facial image, wherein the designated facial image and the designated expression image are the same real person image.

3. A neural network training method, characterized in that: include: Driving a character three-dimensional model according to an expression parameter set and performing rendering to obtain a virtual expression two-dimensional image corresponding to the expression parameter set, wherein the virtual expression two-dimensional image presents a facial expression of a virtual character; Applying a plurality of real-person face two-dimensional images to the virtual expression two-dimensional images corresponding to the facial expressions using a face-swapping neural network to generate a plurality of real-person expression two-dimensional images, wherein the real-person expression two-dimensional images have the same facial expressions as the virtual expression two-dimensional images, the real-person expression two-dimensional images correspond to the expression parameter set, and the real-person expression two-dimensional images have different faces but the same facial expressions; Repeating the above steps, using different sets of expression parameters, to obtain the corresponding two-dimensional images of real-person expressions of different facial expressions; and A neural network is trained according to each of the expression parameter sets and the two-dimensional images of real human expressions corresponding to each of the expression parameter sets.

4. A method for generating animation by tracking facial expressions, characterized in that: include: Driving a first character three-dimensional model and performing rendering according to a plurality of expression parameter sets to obtain a plurality of first virtual expression two-dimensional images corresponding to the expression parameter sets, each of the first virtual expression two-dimensional images presenting a different facial expression of a virtual character; Training a tracking neural network based on the expression parameter sets and the first virtual expression two-dimensional images; Applying a virtual facial 2D image to a target facial 2D image using a face-swapping neural network to generate a second virtual expression 2D image, wherein the virtual facial 2D image is a facial image of the first character 3D model, and the second virtual expression 2D image is a facial image of the first character 3D model having the same facial expression as the target facial 2D image; Inputting the second virtual expression two-dimensional image into the trained tracking neural network to obtain a predicted expression parameter set; and A second character model is controlled using the predicted expression parameter set.

Images