Face driving method, device and terminal
By inputting the to-processed image frames into the face-driven network and superimposing them with preset textures in the face-driven technology, the problem of missing expression details of the three-dimensional model is solved, and the reality of the three-dimensional face model is significantly improved.
Patent Information
- Application Number
- CN202211075883.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-02
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2042-09-02
AI Technical Summary
The existing face driving technology lacks details on the facial expressions of three-dimensional models, resulting in insufficient realism.
By inputting the pending image frames into the face-driven network, the geometric model and texture with the first expression are obtained, and superimposed with the preset expressionless texture, a second texture with rich details is generated, and the geometric model is rendered to the geometric model to improve The reality of the three-dimensional face model.
It improves the richness of the details of the three-dimensional model on the facial expressions and enhances the realistic feeling of the model.
Smart Images

Figure CN117649690B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of computer vision technology, and in particular to a face driving method, device and terminal. Background Art
[0002] Face-driven technology builds a 3D model of a training object on a computer device through a video or photo, and drives the 3D model through the expression of another object, so that the 3D model has both the characteristics of the training object and the expression of the other object. At present, this technology is often applied to the field of film and television entertainment. For example, by capturing the facial expressions of actors and matching them with the expressions of the 3D model on the computer device, a 3D model with the facial expressions of the actors is finally presented to obtain a more realistic picture with the characteristics of the training object.
[0003] Since this technology uses the expression of another object to drive the 3D model, and the 3D model is trained based on part of the training object's data, the expression driving the 3D model may contain expressions that are not in the data for training the 3D model. The lack of such expressions in the training data can easily lead to the lack of details in the facial expressions of the final 3D model, making the 3D model lack of realism. Summary of the invention
[0004] In view of the above problems, the embodiments of the present application provide a face driving method, device and terminal, which can enhance the richness of details of facial expressions in a three-dimensional model, thereby improving the realism of the three-dimensional model.
[0005] In a first aspect, an embodiment of the present application provides a face driving method, comprising:
[0006] Inputting the image frame to be processed into a face drive network to obtain a geometric model with a first expression and a first texture; wherein the face drive network is pre-trained based on a training data set of the target object;
[0007] The first texture and the preset expressionless texture are superimposed to obtain a second texture;
[0008] Based on the second texture rendering geometric model, a three-dimensional face model of the target object is obtained; wherein the three-dimensional face model has a first expression.
[0009] In an optional implementation, the face driving network includes an expression sub-network, a model sub-network, a texture sub-network, a first multi-layer perceptron MLP sub-network, and a second MLP sub-network;
[0010] Inputting the image frame to be processed into the face driving network to obtain a geometric model with a first expression and a first texture, including:
[0011] Input the image frame to be processed into the expression sub-network to obtain the first expression latent vector;
[0012] Inputting the first expression latent vector into the first MLP sub-network to obtain a second expression latent vector; and inputting the first expression latent vector into the second MLP sub-network to obtain a third expression latent vector;
[0013] Inputting the second expression latent vector into the model sub-network to obtain a geometric model with the first expression;
[0014] The third expression latent vector is input into the texture sub-network to obtain the first texture.
[0015] In an optional implementation, before inputting the image frame to be processed into the face drive network, the method further includes:
[0016] Obtain a training data set of the target object; the training data set includes multiple sets of training data, each set of training data includes at least a first image to be trained, a model to be trained, and a corresponding texture to be trained;
[0017] Training the initial expression sub-network according to the first image to be trained to obtain a trained expression sub-network, training the initial model sub-network according to the model to be trained to obtain a trained model sub-network, and training the initial texture sub-network according to the texture to be trained to obtain a trained texture sub-network;
[0018] According to the training data set, the initial first MLP sub-network and the initial second MLP sub-network are trained respectively to obtain the trained first MLP sub-network and the second MLP sub-network;
[0019] The trained expression sub-network, model sub-network, texture sub-network, first MLP sub-network and second MLP sub-network are determined as the face driving network;
[0020] Optimizing the face drive network according to the acquired second image to be trained and the corresponding label information to obtain an optimized face drive network;
[0021] The label information is used to identify the type of the second image to be trained.
[0022] In an optional implementation, the first image to be trained includes a first image frame, a second image frame, and a third image frame; wherein the first image frame has first expression information and first angle information, the second expression information of the second image frame is different from the first expression information, and the second angle information of the third image frame is different from the first angle information;
[0023] The initial expression sub-network is trained according to the first image to be trained to obtain a trained expression sub-network, including:
[0024] Input the first image frame into the initial expression sub-network to obtain a fourth expression latent vector and a first perspective latent vector;
[0025] Input the second image frame into the initial expression sub-network to obtain the second perspective latent vector;
[0026] Input the third image frame into the initial expression sub-network to obtain the fifth expression latent vector;
[0027] The parameters of the initial expression sub-network are updated based on the expression loss function to obtain the trained expression sub-network; the expression loss function is constructed based on the difference between the fourth expression latent vector and the fifth expression latent vector, and the difference between the first perspective latent vector and the second perspective latent vector.
[0028] In an optional implementation, the model to be trained includes: a first model, a second model and a third model; wherein the first model has third expression information and third angle information, the fourth expression information of the second model is different from the third expression information, and the fourth angle information of the third model is different from the third angle information;
[0029] The initial model sub-network is trained according to the model to be trained to obtain a trained model sub-network, including:
[0030] Input the first model, the second model, and the third model into the initial model sub-network respectively to obtain the corresponding fourth model, the fifth model, and the sixth model;
[0031] The parameters of the initial model sub-network are updated based on the model loss function to obtain a trained model sub-network; the model loss function is constructed based on the difference between the fourth model and the first model, the difference between the fifth model and the second model, and the difference between the sixth model and the third model.
[0032] In an optional implementation, the texture to be trained includes a wrinkle map of the first multi-layer texture, and the initial texture sub-network is trained according to the texture to be trained to obtain a trained texture sub-network, including:
[0033] Inputting the wrinkle map of the first multi-layer texture into the initial texture sub-network to obtain the wrinkle map of the second multi-layer texture;
[0034] The parameters of the initial texture sub-network are updated based on the texture loss function to obtain a trained texture sub-network; the texture loss function is constructed based on the difference between the wrinkle map of the first multi-layer texture and the wrinkle map of the second multi-layer texture.
[0035] In an optional implementation, the initial first MLP sub-network and the initial second MLP sub-network are trained according to the training data set to obtain the trained first MLP sub-network and the second MLP sub-network, including:
[0036] Inputting the first image to be trained into the expression sub-network to obtain a first training expression latent vector;
[0037] Input the model to be trained into the model sub-network to obtain the second training expression latent vector;
[0038] Inputting the first training expression latent vector into the initial first MLP sub-network to obtain a third training expression latent vector;
[0039] Based on the first MLP loss function, the parameters of the initial first MLP sub-network are updated to obtain the trained first MLP sub-network; the first MLP loss function is constructed based on the difference between the third training expression latent vector and the second training expression latent vector;
[0040] Input the texture to be trained into the texture sub-network to obtain a fourth training expression latent vector;
[0041] Inputting the first training expression latent vector into the initial second MLP sub-network to obtain a fifth training expression latent vector;
[0042] The parameters of the initial second MLP sub-network are updated based on the second MLP loss function to obtain a trained second MLP sub-network; the second MLP loss function is constructed based on the difference between the fifth training expression latent vector and the fourth training expression latent vector;
[0043] The first image to be trained, the model to be trained and the texture to be trained all have the same angle information and expression information.
[0044] In an optional implementation, optimizing the face drive network according to the acquired second image to be trained and the corresponding label information to obtain the optimized face drive network includes:
[0045] Inputting the second image to be trained into the face drive network to obtain a face drive image;
[0046] The parameters of the face drive network are updated based on the modeling loss function to obtain an optimized face drive network; wherein the modeling loss function is constructed based on the difference between the face drive image and the second image to be trained.
[0047] In a second aspect, an embodiment of the present application provides a face driving device, comprising:
[0048] An image processing module, used for inputting the image frame to be processed into a face drive network to obtain a geometric model with a first expression and a first texture; wherein the face drive network is pre-trained based on a training data set of a target object;
[0049] A texture processing module, used for superimposing the first texture and the preset expressionless texture to obtain a second texture;
[0050] The model rendering module is used to render the geometric model based on the second texture to obtain a three-dimensional face model of the target object; wherein the three-dimensional face model has a first expression.
[0051] In a third aspect, an embodiment of the present application provides a terminal, comprising: a memory and a processor; the memory stores a computer program or instruction that can be executed by the processor; the processor executes the computer program or instruction to implement any one of the face driving methods in the first aspect above.
[0052] In a fourth aspect, an embodiment of the present application provides a computer storage medium, in which a computer program or instruction that can be executed by a processor is stored. When the computer program or instruction is executed, the face driving method of any one of the above-mentioned first aspects is implemented.
[0053] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes the face driving method described in any one of the above-mentioned first aspects.
[0054] The face driving method, device and terminal provided in the embodiments of the present application input the first expression carried by the driving object into the face driving network of the target object through the image frame to be processed of the driving object, thereby driving the face driving network to output a first texture and a geometric model belonging to the target object with the first expression, and obtain a second texture of the target object with rich details of the first expression by superimposing the first texture of the target object carrying the texture features of the first expression with a preset expressionless texture with rich texture details, and render the second texture to the geometric model to obtain a three-dimensional face model of the target object with the first expression, and ensure that the texture details of the three-dimensional face model are rich and have a strong sense of reality.
[0055] The above description is only an overview of the technical solution of the embodiment of the present application. In order to more clearly understand the technical means of the embodiment of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] In order to more clearly illustrate the technical solutions of the embodiments of the present application, a brief introduction will be given below to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0057] Figure 1 It is a flow chart of the face driving method provided in the embodiment of the present application.
[0058] Figure 2 It is a structural diagram of a face drive network provided in an embodiment of the present application.
[0059] Figure 3 It is a structural diagram of an expression sub-network provided in an embodiment of the present application.
[0060] Figure 4 It is a structural diagram of a model sub-network provided in an embodiment of the present application.
[0061] Figure 5 It is a structural diagram of a texture sub-network provided in an embodiment of the present application.
[0062] Figure 6 It is a structural diagram of a training MLP sub-network provided in an embodiment of the present application.
[0063] Figure 7 It is a structural schematic diagram of a face driving device provided in an embodiment of the present application.
[0064] Figure 8 It is a terminal provided in an embodiment of the present application. DETAILED DESCRIPTION
[0065] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs; the terms used in the specification of the application are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" and any variations thereof in the specification, claims and drawings of this application are intended to cover non-exclusive inclusions.
[0067] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase "embodiments" in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0068] The term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists, A and B exist at the same time, and B exists. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.
[0069] The directional words appearing in the following description are all directions shown in the drawings, and are not specific limitations of the present application. For example, in the description of the present application, the directions or positional relationships indicated by the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc. are based on the directions or positional relationships shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as a limitation of the present application.
[0070] In addition, the terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects rather than to describe a specific order, and may explicitly or implicitly include one or more of the features.
[0071] In the description of the present application, unless otherwise specified, "plurality" means more than two (including two), and similarly, "multiple groups" means more than two groups (including two).
[0072] Face-driven technology builds a 3D model of a training object on a computer device through a video or image, and drives the 3D model through the expression of another object, so that the 3D model has both the characteristics of the training object and the expression of the other object. It is usually used when the movements of the training object are difficult to obtain or difficult to perform as required. For example, this technology is applied to the field of film and television entertainment. By capturing the facial expressions of the actors, it is matched with the expressions of the 3D model of the training object on the computer device, and finally presents the 3D model of the training object with the facial expressions of the actors, so as to obtain a more realistic picture with the characteristics of the training object.
[0073] Since this technology uses the expression of another object to drive the 3D model, and the 3D model is trained based on part of the training object's data, the expression driving the 3D model may contain expressions that are not in the data for training the 3D model. The lack of such expressions in the training data can easily lead to the lack of details in the facial expressions of the final 3D model, making the 3D model lack of realism.
[0074] Based on the above problems, the embodiments of the present application provide a face driving method, device and terminal, which can enhance the richness of details of facial expressions in a three-dimensional model, thereby enhancing the realism of the three-dimensional model.
[0075] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.
[0076] Figure 1 FIG. 1 is a flow chart of a face driving method provided in an embodiment of the present application. Figure 1 As shown, the face driving method provided in the embodiment of the present application may include:
[0077] S101, inputting the image frame to be processed into a face drive network to obtain a geometric model with a first expression and a first texture. The face drive network is pre-trained based on a training data set of a target object.
[0078] Optionally, the image frame to be processed is an image frame of a driving object, and the driving object may be a target object or other driving object used to drive a face driving network. Specifically, the image frame to be processed may be obtained from a video.
[0079] The image frame to be processed contains the first expression information of the driving object. After the image frame to be processed is input into the face driving network of the target object, the obtained geometric model and the first texture of the target object both have the first expression.
[0080] S102: Superimpose the first texture and the preset expressionless texture to obtain a second texture.
[0081] Specifically, the preset expressionless texture is a texture corresponding to a preset expressionless image frame with a higher resolution, which has more texture details. The first texture is the difference between the texture carried in the image frame to be processed and the preset expressionless texture, so as to extract more obvious texture details in the image frame to be processed.
[0082] Exemplarily, the resolution of the first texture is 512 pixels × 512 pixels, and the resolution of the preset expressionless texture is 4K. In the face-driven network, the 4K resolution (4096 pixels × 4096 pixels) of the preset expressionless texture is reduced to 512 pixels × 512 pixels, which is the same as the resolution of the image frame to be processed, so as to make a difference at the corresponding pixel points to obtain a first texture with a resolution of 512 pixels × 512 pixels. Then, the first texture with a resolution of 512 pixels × 512 pixels is linearly enlarged to obtain an enlarged version of the first texture with a resolution of 4K, and the enlarged version of the first texture is superimposed on the preset expressionless texture that has not been reduced to obtain a second texture.
[0083] In the face-driven network, using a first texture with a smaller resolution can reduce the amount of computation of the face-driven network. The image size processed in the face-driven network is 512 pixels × 512 pixels, which reduces the resolution requirement for the input image frame to be processed. After the face-driven network predicts the first texture, by superimposing the first texture on a high-resolution preset expressionless texture, the high-resolution preset expressionless texture can be used to enrich the details of the first texture, making the obtained second texture more realistic.
[0084] S103: Rendering the geometric model based on the second texture to obtain a three-dimensional face model of the target object, wherein the three-dimensional face model has a first expression.
[0085] A more realistic second texture is rendered onto the geometric model of the target object. Since both the second texture and the geometric model have the first expression, the faces represented by the second model and the geometric model match. By rendering the geometric model using the second texture, a three-dimensional face model of the target object with rich texture details can be obtained. The three-dimensional face model has the first expression carried by the image frame to be processed of the driving object, thereby obtaining a three-dimensional face model of the target object presenting the first expression.
[0086] The face driving method provided in the embodiment of the present application can input the first expression carried by the driving object into the face driving network of the target object through the image frame to be processed of the driving object, thereby driving the face driving network to output a first texture and a geometric model belonging to the target object with the first expression, and superimpose the first texture of the target object carrying the texture features of the first expression with a preset expressionless texture with rich texture details to obtain a second texture of the target object with rich details of the first expression, and render the second texture to the geometric model to obtain a three-dimensional face model of the target object with the first expression, and ensure that the texture details of the three-dimensional face model are rich and have a strong sense of reality.
[0087] Optionally, Figure 2 Schematic diagram of a face driving network provided in an embodiment of the present application. Figure 2 As shown, the face driving network in S101 may include: an expression sub-network, a model sub-network, a texture sub-network, a first multi-layer perceptron MLP sub-network, and a second MLP sub-network. Figure 2 , S101 may include:
[0088] S1011. Input the image frame to be processed into the expression sub-network to obtain a first expression latent vector.
[0089] Specifically, the image frame to be processed is input into the encoder of the expression sub-network, feature information of the first expression of the image frame to be processed is extracted, and a first expression latent vector having the first expression feature is obtained.
[0090] S1012. Input the first expression latent vector into the first MLP sub-network to obtain the second expression latent vector, and input the first expression latent vector into the second MLP sub-network to obtain the third expression latent vector.
[0091] Specifically, the first MLP subnetwork and the second MLP subnetwork have different functions. The first MLP subnetwork can further extract features related to the geometric model of the face from the first expression latent vector to obtain a second expression latent vector with model features. The second MLP subnetwork can further extract features related to the texture of the face from the first expression latent vector to obtain a third expression latent vector with texture features.
[0092] S1013. Input the second expression latent vector into the model sub-network to obtain a geometric model with the first expression.
[0093] Since the second expression latent vector is obtained by processing the first expression latent vector of the first expression feature, the second expression latent vector has the feature information of the first expression. The second expression latent vector having both the model feature and the first expression feature is input into the model sub-network, and a geometric model having the first expression is obtained after processing.
[0094] S1014. Input the third expression latent vector into the texture sub-network to obtain the first texture.
[0095] Similarly, the third expression latent vector is obtained by processing the first expression latent vector of the first expression feature. Therefore, the third expression latent vector has the feature information of the first expression. The third expression latent vector with both texture features and features of the first expression is input into the texture sub-network, and the first texture with the first expression is obtained after processing.
[0096] Optionally, before S101, the face driving method improved by the embodiment of the present application may further include:
[0097] S201, obtaining a training data set of a target object, wherein the training data set includes multiple groups of training data, and each group of training data includes at least a first image to be trained, a model to be trained, and a corresponding texture to be trained.
[0098] Optionally, the target object may be photographed at a high frequency by a high-speed camera to obtain multiple first images to be trained of the target object within a short period of time, so as to ensure that the first images to be trained have dynamic details and detail changes of the target object with a high frame rate.
[0099] S202: Train the initial expression sub-network according to the first image to be trained to obtain a trained expression sub-network.
[0100] Specifically, the first to-be-trained image of the target object is used to train the expression sub-network of the target object.
[0101] Optionally, the first image to be trained includes a first image frame, a second image frame and a third image frame; wherein the first image frame has first expression information and first angle information, the second expression information of the second image frame is different from the first expression information, and the second angle information of the third image frame is different from the first angle information.
[0102] Specifically, when the expression of the target object is the first expression, the camera shoots the target object at the first angle to obtain the first image frame, and at the same time, another camera shoots the target object at the second angle to obtain the third image frame. At another time, the expression of the target object changes to the second expression, and at this time, the camera shoots the target object at the first angle to obtain the second image frame. That is, the first image frame and the second image frame have the same viewing angle information and different expression information, and the first image frame and the third image frame have the same expression information and different viewing angle information.
[0103] Figure 3 Schematic diagram of the structure of an expression sub-network provided in an embodiment of the present application. Figure 3 As shown, S202 may include:
[0104] S2021. Input the first image frame into the initial expression sub-network to obtain a fourth expression latent vector and a first perspective latent vector.
[0105] By inputting the first image frame having the first expression information and the first angle information into the expression sub-network, a fourth expression latent vector having the first expression information and a first perspective latent vector having the first angle information can be obtained.
[0106] S2022. Input the second image frame into the initial expression sub-network to obtain a second perspective latent vector.
[0107] Specifically, a second image frame having the same perspective information as the first image frame but different expression information is input into the expression sub-network, and a second perspective latent vector of the first perspective information can be obtained.
[0108] S2023. Input the third image frame into the initial expression sub-network to obtain a fifth expression latent vector.
[0109] Specifically, a third image frame having the same expression information as the first image frame but different viewing angle information is input into the expression sub-network, so as to obtain a fifth expression latent vector of the first expression information.
[0110] S2024, updating the parameters of the initial expression sub-network based on the expression loss function to obtain a trained expression sub-network. The expression loss function is constructed based on the difference between the fourth expression latent vector and the fifth expression latent vector, and the difference between the first view latent vector and the second view latent vector.
[0111] An expression loss function is constructed by using the difference between the fourth expression latent vector and the fifth expression latent vector of the first expression information and the difference between the first perspective latent vector and the second perspective latent vector of the first perspective information, and the expression sub-network is trained so that the finally trained expression sub-network can correctly parse the expression and perspective of the image frame.
[0112] S203: Train the initial model sub-network according to the model to be trained to obtain a trained model sub-network.
[0113] Specifically, the to-be-trained model of the target object is used to train the model sub-network of the target object.
[0114] Optionally, the model to be trained includes: a first model, a second model and a third model; wherein the first model has third expression information and third angle information, the fourth expression information of the second model is different from the third expression information, and the fourth angle information of the third model is different from the third angle information.
[0115] Specifically, the first model is a model of the target object at the third angle when presenting the third expression, the second model is a model of the target object at the third angle when presenting the fourth expression, and the third model is a model of the target object at the fourth angle when presenting the third expression. That is, the first model and the second model have the same perspective information but different expression information, and the first model and the third model have the same expression information but different perspective information.
[0116] Figure 4 Schematic diagram of a model sub-network provided in an embodiment of the present application. Figure 4 As shown, S203 may include:
[0117] S2031. Input the first model, the second model, and the third model into the initial model sub-network respectively to obtain the corresponding fourth model, the fifth model, and the sixth model.
[0118] S2032: Update the parameters of the initial model sub-network based on the model loss function to obtain a trained model sub-network. The model loss function is constructed based on the difference between the fourth model and the first model, the difference between the fifth model and the second model, and the difference between the sixth model and the third model.
[0119] The model loss function is constructed by the difference between the fourth model and the first model, the difference between the fifth model and the second model, and the difference between the sixth model and the third model, and the model sub-network is trained so that the finally trained model sub-network can correctly parse the input model.
[0120] S204: Train the initial texture sub-network according to the texture to be trained to obtain a trained texture sub-network.
[0121] Specifically, the texture to be trained of the target object is used to train the texture sub-network of the target object.
[0122] Optionally, the texture to be trained includes a wrinkle map of the first multi-layer texture. Figure 5 Schematic diagram of a texture sub-network provided in an embodiment of the present application. Figure 5 As shown, S204 may include:
[0123] S2041. Input the wrinkle map of the first multi-layer texture into the initial texture sub-network to obtain the wrinkle map of the second multi-layer texture.
[0124] S2042: Update the parameters of the initial texture sub-network based on the texture loss function to obtain a trained texture sub-network. The texture loss function is constructed based on the difference between the wrinkle map of the first multi-layer texture and the wrinkle map of the second multi-layer texture.
[0125] The texture loss function is constructed by the difference between the wrinkle map of the first multi-layer texture and the wrinkle map of the second multi-layer texture, and the texture sub-network is trained so that the finally trained texture sub-network can correctly parse the input texture.
[0126] S205 . Train the initial first MLP sub-network and the initial second MLP sub-network respectively according to the training data set to obtain the trained first MLP sub-network and the trained second MLP sub-network.
[0127] Specifically, the first MLP sub-network and the second MLP sub-network are trained according to the one-to-one corresponding first images to be trained, models to be trained, and textures to be trained in the training data set.
[0128] Optionally, Figure 6 Schematic diagram of a structure of a training MLP sub-network provided in an embodiment of the present application. Figure 6 As shown, S205 may include:
[0129] S2051. Input the first image to be trained into the expression sub-network to obtain a first training expression latent vector.
[0130] The first image to be trained is input into the expression sub-network trained according to the first image to be trained in S202 to obtain a first training expression latent vector.
[0131] S2052. Input the model to be trained into the model sub-network to obtain a second training expression latent vector.
[0132] The model to be trained corresponding to the first image to be trained is input into S203 according to the model sub-network trained according to the model to be trained, so as to obtain a second training expression latent vector.
[0133] S2053. Input the first training expression latent vector into the initial first MLP sub-network to obtain a third training expression latent vector.
[0134] S2054: Update the parameters of the initial first MLP sub-network based on the first MLP loss function to obtain a trained first MLP sub-network. The first MLP loss function is constructed based on the difference between the third training expression latent vector and the second training expression latent vector.
[0135] The first MLP loss function is constructed by the difference between the third training expression latent vector and the second training expression latent vector, and the first MLP sub-network is trained so that the finally trained first MLP sub-network can extract the model features from the input expression latent vector to obtain the expression latent vector with model features.
[0136] S2055. Input the texture to be trained into the texture sub-network to obtain a fourth training expression latent vector.
[0137] The texture to be trained corresponding to the first image to be trained is input into the texture sub-network trained according to the texture to be trained in S204 to obtain a fourth training expression latent vector.
[0138] S2056. Input the first training expression latent vector into the initial second MLP sub-network to obtain the fifth training expression latent vector.
[0139] S2057, based on the second MLP loss function, update the parameters of the initial second MLP sub-network to obtain a trained second MLP sub-network. The second MLP loss function is constructed based on the difference between the fifth training expression latent vector and the fourth training expression latent vector. The first image to be trained, the model to be trained, and the texture to be trained have the same angle information and expression information.
[0140] The second MLP loss function is constructed by the difference between the fifth training expression latent vector and the fourth training expression latent vector, and the second MLP sub-network is trained so that the finally trained second MLP sub-network can extract the texture features in the input expression latent vector to obtain the expression latent vector with texture features.
[0141] S206: Determine the trained expression sub-network, model sub-network, texture sub-network, first MLP sub-network and second MLP sub-network as a face driving network.
[0142] S207: Optimize the face drive network according to the acquired second image to be trained and the corresponding label information to obtain an optimized face drive network, wherein the label information is used to identify the type of the second image to be trained.
[0143] It is understandable that the second image to be trained may be a video frame of the target object or a video frame of the driving object, and the shooting environment of the second image to be trained may be different from the shooting environment in the training data set of the target object. Therefore, when the shooting environments are different, or the target object is different from the driving object, before the face driving network is driven by the second image to be trained of the driving object for modeling, the face driving network needs to be optimized by part of the second image to be trained, and optimized to a state where the face driving network can be accurately driven by the object and / or environment in the second image to be trained, so as to reduce the modeling error caused by different objects or environments.
[0144] Specifically, the label information is represented by an indicator vector, and after part of the second image to be trained is input into the face drive network, the face drive network optimizes the part of the face drive network related to the second image to be trained according to the indicator vector. Optionally, the indicator vector can be composed of two values 1 and / or 0. During optimization, only the part of the network corresponding to 1 is activated and optimized, so that the optimized face drive network can ignore the difference between the environment or object during driving and training, and obtain a better driving effect.
[0145] Optionally, S207 may include:
[0146] S2071. Input the second image to be trained into the face drive network to obtain a face drive image.
[0147] For details, reference may be made to the process of training the face drive network according to the training data set of the target object in the above embodiment, which will not be described in detail here.
[0148] S2072: Update the parameters of the face drive network based on the modeling loss function to obtain an optimized face drive network, wherein the modeling loss function is constructed based on the difference between the face drive image and the second image to be trained.
[0149] A modeling loss function is constructed by the difference between the face driving image and the second image to be trained, and the parameters of the face driving network are optimized so that the optimized face driving network can accurately extract the expression features of the input driving object, so that it can combine the expression of the driving object with the model of the target object, and accurately predict the texture of the face of the target object under the expression, thereby obtaining a three-dimensional face model of the target object with rich details and strong sense of reality.
[0150] Figure 7 Schematic diagram of the structure of a face driving device provided in an embodiment of the present application. Figure 7 As shown, the face driving device provided in the embodiment of the present application may include:
[0151] The image processing module 701 is used to input the image frame to be processed into a face drive network to obtain a geometric model with a first expression and a first texture; wherein the face drive network is pre-trained based on a training data set of a target object.
[0152] The texture processing module 702 is used to superimpose the first texture and the preset expressionless texture to obtain a second texture.
[0153] The model rendering module 703 is used to render the geometric model based on the second texture to obtain a three-dimensional face model of the target object; wherein the three-dimensional face model has a first expression.
[0154] Optionally, the face driving network includes an expression sub-network, a model sub-network, a texture sub-network, a first multi-layer perceptron MLP sub-network and a second MLP sub-network; the image processing module 701 is specifically used to: input the image frame to be processed into the expression sub-network to obtain a first expression latent vector; input the first expression latent vector into the first MLP sub-network to obtain a second expression latent vector; and input the first expression latent vector into the second MLP sub-network to obtain a third expression latent vector; input the second expression latent vector into the model sub-network to obtain a geometric model with a first expression; and input the third expression latent vector into the texture sub-network to obtain a first texture.
[0155] Optionally, before the image frame to be processed is input into the face driving network, the face driving device also includes: a data acquisition module 704, which is used to obtain a training data set of the target object; the training data set includes multiple groups of training data, and each group of training data includes at least a first image to be trained, a model to be trained, and a corresponding texture to be trained; an expression sub-network training module 705, which is used to train the initial expression sub-network according to the first image to be trained to obtain a trained expression sub-network; a model sub-network training module 706, which is used to train the initial model sub-network according to the model to be trained to obtain a trained model sub-network; and a texture sub-network training module 707, which is used to train the initial texture sub-network according to the texture to be trained. The training is performed to obtain a trained texture sub-network; an MLP sub-network training module 708 is used to train the initial first MLP sub-network and the initial second MLP sub-network according to the training data set, respectively, to obtain the trained first MLP sub-network and the second MLP sub-network; a modeling network determination module 709 is used to determine the trained expression sub-network, model sub-network, texture sub-network, first MLP sub-network and second MLP sub-network as a face drive network; a network optimization module 710 is used to optimize the face drive network according to the acquired second image to be trained and the corresponding label information, to obtain an optimized face drive network; wherein the label information is used to identify the type of the second image to be trained.
[0156] Optionally, the first image to be trained includes a first image frame, a second image frame and a third image frame; wherein the first image frame has first expression information and first angle information, the second expression information of the second image frame is different from the first expression information, and the second angle information of the third image frame is different from the first angle information; the expression sub-network training module 705 is specifically used to: input the first image frame into the initial expression sub-network to obtain a fourth expression latent vector and a first perspective latent vector; input the second image frame into the initial expression sub-network to obtain a second perspective latent vector; input the third image frame into the initial expression sub-network to obtain a fifth expression latent vector; update the parameters of the initial expression sub-network based on the expression loss function to obtain a trained expression sub-network; the expression loss function is constructed based on the difference between the fourth expression latent vector and the fifth expression latent vector, and the difference between the first perspective latent vector and the second perspective latent vector.
[0157] Optionally, the models to be trained include: a first model, a second model and a third model; wherein the first model has third expression information and third angle information, the fourth expression information of the second model is different from the third expression information, and the fourth angle information of the third model is different from the third angle information; the model sub-network training module 706 is specifically used to: input the first model, the second model, and the third model into the initial model sub-network respectively to obtain the corresponding fourth model, the fifth model, and the sixth model; based on the model loss function, update the parameters of the initial model sub-network to obtain a trained model sub-network; the model loss function is constructed based on the difference between the fourth model and the first model, the difference between the fifth model and the second model, and the difference between the sixth model and the third model.
[0158] Optionally, the texture to be trained includes a wrinkle map of a first multi-layer texture, and the texture sub-network training module 707 is specifically used to: input the wrinkle map of the first multi-layer texture into an initial texture sub-network to obtain a wrinkle map of a second multi-layer texture; update the parameters of the initial texture sub-network based on a texture loss function to obtain a trained texture sub-network; the texture loss function is constructed based on the difference between the wrinkle map of the first multi-layer texture and the wrinkle map of the second multi-layer texture.
[0159] Optionally, the MLP subnetwork training module 708 is specifically used to: input the first image to be trained into the expression subnetwork to obtain a first training expression latent vector; input the model to be trained into the model subnetwork to obtain a second training expression latent vector; input the first training expression latent vector into the initial first MLP subnetwork to obtain a third training expression latent vector; update the parameters of the initial first MLP subnetwork based on the first MLP loss function to obtain a trained first MLP subnetwork; the first MLP loss function is constructed based on the difference between the third training expression latent vector and the second training expression latent vector; input the texture to be trained into the texture subnetwork to obtain a fourth training expression latent vector; input the first training expression latent vector into the initial second MLP subnetwork to obtain a fifth training expression latent vector; update the parameters of the initial second MLP subnetwork based on the second MLP loss function to obtain a trained second MLP subnetwork; the second MLP loss function is constructed based on the difference between the fifth training expression latent vector and the fourth training expression latent vector; wherein the first image to be trained, the model to be trained and the texture to be trained have the same angle information and expression information.
[0160] Optionally, the network optimization module 710 is specifically used to: input the second image to be trained into the face drive network to obtain a face drive image; update the parameters of the face drive network based on the modeling loss function to obtain an optimized face drive network; wherein the modeling loss function is constructed based on the difference between the face drive image and the second image to be trained.
[0161] Figure 8 This is a terminal 1 provided in an embodiment of the present application. Figure 8 As shown, the terminal provided in the embodiment of the present application may include: a memory 11 and a processor 10; the memory 11 stores a computer program 12 or instructions that can be executed by the processor; the processor 10 executes the computer program 12 or the instructions to implement the face driving method described in any one of the above method embodiments.
[0162] Optionally, the terminal 1 provided in the embodiment of the present application may further include a display 13 for displaying a three-dimensional face model obtained by the face driving method.
[0163] In summary, the face driving method, device and terminal provided in the embodiments of the present application input the first expression carried by the driving object into the face driving network of the target object through the image frame to be processed of the driving object, thereby driving the face driving network to output a first texture and a geometric model belonging to the target object with the first expression, and superimpose the first texture of the target object carrying the texture features of the first expression with the preset expressionless texture rich in texture details to obtain the second texture of the target object with rich details of the first expression, and render the second texture to the geometric model to obtain a three-dimensional face model of the target object with the first expression, and ensure that the texture details of the three-dimensional face model are rich and have a strong sense of reality.
[0164] An embodiment of the present application also provides a computer storage medium, in which a computer program or instruction that can be executed by a processor is stored. When the computer program or instruction is executed, the face driving method of any one of the above method embodiments is implemented.
[0165] An embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes the face driving method described in any one of the above method embodiments.
[0166] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0167] Each functional unit or module in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional unit.
[0168] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program code.
[0169] In the claims, any reference symbols placed between brackets shall not be construed as limiting the claims. The word "comprising" described in the present application does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application can be implemented with the aid of hardware comprising several different elements and with the aid of a suitably programmed computer. In a unit claim that lists a number of devices, several units of these devices may be embodied by the same hardware item. The use of first, second, and third, etc. does not indicate any order, and these words may be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be understood as limitations on the order of execution.
[0170] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A face driving method, characterized in that: The method comprises: Inputting the image frame to be processed into a face drive network to obtain a geometric model with a first expression and a first texture; wherein the face drive network is pre-trained based on a training data set of the target object; Superimposing the first texture and the preset expressionless texture to obtain a second texture; Rendering the geometric model based on the second texture to obtain a three-dimensional face model of the target object; wherein the three-dimensional face model has the first expression; wherein the face driving network includes an expression sub-network, a model sub-network, a texture sub-network, a first multi-layer perceptron MLP sub-network and a second MLP sub-network; The step of inputting the image frame to be processed into a face driving network to obtain a geometric model with a first expression and a first texture includes: Inputting the image frame to be processed into the expression sub-network to obtain a first expression latent vector; Inputting the first expression latent vector into the first MLP sub-network to obtain a second expression latent vector; and inputting the first expression latent vector into the second MLP sub-network to obtain a third expression latent vector; Inputting the second expression latent vector into the model sub-network to obtain the geometric model with the first expression; The third expression latent vector is input into the texture sub-network to obtain the first texture.
2. The method according to claim 1, characterized in that Before inputting the image frame to be processed into the face drive network, the method further includes: Acquire a training data set of the target object; the training data set includes multiple sets of training data, each set of training data includes at least a first image to be trained, a model to be trained, and a corresponding texture to be trained; Training the initial expression sub-network according to the first image to be trained to obtain the trained expression sub-network, training the initial model sub-network according to the model to be trained to obtain the trained model sub-network, and training the initial texture sub-network according to the texture to be trained to obtain the trained texture sub-network; According to the training data set, the initial first MLP sub-network and the initial second MLP sub-network are trained respectively to obtain the trained first MLP sub-network and the second MLP sub-network; Determine the trained expression sub-network, the model sub-network, the texture sub-network, the first MLP sub-network and the second MLP sub-network as the face driving network; Optimizing the face drive network according to the acquired second to-be-trained image of the target object and the corresponding label information to obtain the optimized face drive network; The label information is used to identify the type of the second image to be trained.
3. The method according to claim 2, characterized in that The first image to be trained includes a first image frame, a second image frame and a third image frame; wherein the first image frame has first expression information and first angle information, the second expression information of the second image frame is different from the first expression information, and the second angle information of the third image frame is different from the first angle information; Training the initial expression sub-network according to the first image to be trained to obtain the trained expression sub-network includes: Inputting the first image frame into the initial expression sub-network to obtain a fourth expression latent vector and a first perspective latent vector; Inputting the second image frame into the initial expression sub-network to obtain a second perspective latent vector; Inputting the third image frame into the initial expression sub-network to obtain a fifth expression latent vector; The parameters of the initial expression sub-network are updated based on the expression loss function to obtain the trained expression sub-network; the expression loss function is constructed based on the difference between the fourth expression latent vector and the fifth expression latent vector, and the difference between the first perspective latent vector and the second perspective latent vector.
4. The method according to claim 2, characterized in that: The model to be trained includes: a first model, a second model and a third model; wherein the first model has third expression information and third angle information, the fourth expression information of the second model is different from the third expression information, and the fourth angle information of the third model is different from the third angle information; The initial model sub-network is trained according to the model to be trained to obtain the trained model sub-network, including: Inputting the first model, the second model, and the third model into the initial model sub-network respectively to obtain the corresponding fourth model, the fifth model, and the sixth model; The parameters of the initial model sub-network are updated based on the model loss function to obtain the trained model sub-network; the model loss function is constructed based on the difference between the fourth model and the first model, the difference between the fifth model and the second model, and the difference between the sixth model and the third model.
5. The method according to claim 2, characterized in that: The texture to be trained includes a wrinkle map of a first multi-layer texture, and the initial texture sub-network is trained according to the texture to be trained to obtain the trained texture sub-network, including: Inputting the wrinkle map of the first multi-layer texture into the initial texture sub-network to obtain the wrinkle map of the second multi-layer texture; The parameters of the initial texture sub-network are updated based on a texture loss function to obtain the trained texture sub-network; the texture loss function is constructed based on the difference between the wrinkle map of the first multi-layer texture and the wrinkle map of the second multi-layer texture.
6. The method according to claim 3, characterized in that The step of training the initial first MLP sub-network and the initial second MLP sub-network respectively according to the training data set to obtain the trained first MLP sub-network and the trained second MLP sub-network comprises: Inputting the first image to be trained into the expression sub-network to obtain a first training expression latent vector; Inputting the model to be trained into the model sub-network to obtain a second training expression latent vector; Inputting the first training expression latent vector into the initial first MLP sub-network to obtain a third training expression latent vector; Based on the first MLP loss function, the parameters of the initial first MLP sub-network are updated to obtain the trained first MLP sub-network; the first MLP loss function is constructed based on the difference between the third training expression latent vector and the second training expression latent vector; Inputting the texture to be trained into the texture sub-network to obtain a fourth training expression latent vector; Inputting the first training expression latent vector into the initial second MLP sub-network to obtain a fifth training expression latent vector; Based on the second MLP loss function, the parameters of the initial second MLP sub-network are updated to obtain the trained second MLP sub-network; the second MLP loss function is constructed based on the difference between the fifth training expression latent vector and the fourth training expression latent vector; The first image to be trained, the model to be trained and the texture to be trained all have the same angle information and expression information.
7. The method according to claim 2, characterized in that Optimizing the face drive network according to the acquired second image to be trained and the corresponding label information to obtain the optimized face drive network, including: Inputting the second image to be trained into the face drive network to obtain a face drive image; The parameters of the face drive network are updated based on a modeling loss function to obtain the optimized face drive network; wherein the modeling loss function is constructed based on the difference between the face drive image and the second image to be trained.
8. A face driving device, characterized in that: The device comprises: An image processing module, used for inputting the image frame to be processed into a face drive network to obtain a geometric model with a first expression and a first texture; wherein the face drive network is pre-trained based on a training data set of a target object; A texture processing module, used for superimposing the first texture and a preset expressionless texture to obtain a second texture; A model rendering module, configured to render the geometric model based on the second texture to obtain a three-dimensional face model of the target object; wherein the three-dimensional face model has the first expression; The face driving network includes an expression sub-network, a model sub-network, a texture sub-network, a first multi-layer perceptron MLP sub-network and a second MLP sub-network; The step of inputting the image frame to be processed into a face driving network to obtain a geometric model with a first expression and a first texture includes: Inputting the image frame to be processed into the expression sub-network to obtain a first expression latent vector; Inputting the first expression latent vector into the first MLP sub-network to obtain a second expression latent vector; and inputting the first expression latent vector into the second MLP sub-network to obtain a third expression latent vector; Inputting the second expression latent vector into the model sub-network to obtain the geometric model with the first expression; The third expression latent vector is input into the texture sub-network to obtain the first texture.
9. A terminal, characterized in that: The terminal comprises: a memory and a processor; the memory stores a computer program or an instruction that can be executed by the processor; the processor executes the computer program or the instruction to implement the face driving method according to any one of claims 1 to 7.
