Face image generation method, apparatus, device, and storage medium

Abstract

The application discloses a face image generation method, device and equipment and a storage medium, and belongs to the technical field of computers. The method comprises the following steps: acquiring a face image of a first face, acquiring a face image of a second face, and acquiring face vertex data of multiple faces with different face shapes and different expressions; taking the first face as a fitting target, fitting the face vertex data of the multiple faces to determine target face shape features and target expression features of the first face; taking the second face as a fitting target, fitting the face vertex data of the multiple faces to determine target face shape features and target expression features of the second face; and generating a target face image of the second face based on the target expression features of the first face, the target face shape features of the second face, and target face texture features of the second face. The method reduces the time cost and labor cost.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, device, and storage medium for generating face images. Background Technology

[0002] With the continuous development of computer technology, virtual object-driven programming has been widely used in many fields and has great market value. For example, in film production, virtual objects are added to the film, driving these virtual objects to complete corresponding plot tasks.

[0003] Currently, driving a virtual object requires constructing a 3D model of the virtual object, and having professional actors wear motion capture equipment to perform corresponding actions. The motion capture equipment records the motion data of each action, and this motion data is mapped into the 3D model of the virtual object to drive the virtual object.

[0004] Because the facial motion capture of virtual objects requires a high degree of precision, it is necessary to acquire detailed facial motion data and build detailed virtual object facial models. However, acquiring detailed facial motion data requires expensive motion capture equipment, and building detailed virtual object facial models requires a long time cost and a large manpower cost, which cannot meet the current demand for low cost and rapid iteration. Summary of the Invention

[0005] This application provides a method, apparatus, device, and storage medium for generating facial images, which significantly reduces time and labor costs. The technical solution is as follows:

[0006] In one aspect, a method for generating a face image is provided, the method comprising:

[0007] The process involves acquiring a first face image, a second face image, and face vertex data, wherein the face vertex data includes face vertex data of multiple faces with different face shapes and expressions.

[0008] The first face shape feature and the first expression feature are used as weighting coefficients to fit the face vertex data of the multiple faces. The first face is used as the fitting target, and the first face shape feature and the first expression feature are iteratively updated so that the fitting result is close to the first face. The first face shape feature and the first expression feature obtained in the last update are determined as the target face shape feature and target expression feature of the first face, respectively.

[0009] Using the second face shape feature and the second expression feature as weighting coefficients, the face vertex data of the multiple faces are fitted. Taking the second face as the fitting target, the second face shape feature and the second expression feature are iteratively updated so that the fitting result is close to the second face. The second face shape feature and the second expression feature obtained in the last update are determined as the target face shape feature and the target expression feature of the second face, respectively.

[0010] Based on the target expression features of the first face, the target face shape features of the second face, and the target face texture features of the second face, a target face image of the second face is generated.

[0011] In one possible implementation, the step of using the first face shape feature and the first expression feature as weighting coefficients to fit the face vertex data of the plurality of faces, and iteratively updating the first face shape feature and the first expression feature with the first face as the fitting target, so that the fitting result approximates the first face, includes:

[0012] Based on the face image of the first face, obtain the face vertex data of the first face;

[0013] The preset first face shape feature and preset first expression feature are used as weighting coefficients to fit the face vertex data of the multiple faces;

[0014] Based on the fitting results and the face vertex data of the first face, the fitting error is determined;

[0015] The preset first face shape features and the preset first expression features are iteratively updated to reduce the fitting error.

[0016] In one possible implementation, the vertex data of the first face is two-dimensional data, and the vertex data of the plurality of faces is three-dimensional data; determining the fitting error based on the fitting result and the vertex data of the first face includes:

[0017] The three-dimensional face vertex data in the fitting result is projected onto a two-dimensional coordinate system to obtain two-dimensional face vertex data;

[0018] The difference between the obtained two-dimensional face vertex data and the face vertex data of the first face is determined as the fitting error.

[0019] In one possible implementation, the face vertex data of the first face is the face key point data of the first face; the step of fitting the face vertex data of the plurality of faces with preset first face shape features and preset first expression features as weighting coefficients includes:

[0020] Based on the facial key point data of the first face, determine the facial key point data of the plurality of faces that match the facial key point data of the first face;

[0021] The preset first face shape feature and the preset first expression feature are used as weighting coefficients to fit the facial key point data of the multiple faces.

[0022] In one possible implementation, the step of using the first face shape feature and the first expression feature as weighting coefficients to fit the face vertex data of the plurality of faces, taking the first face as the fitting target, iteratively updating the first face shape feature and the first expression feature to make the fitting result approximate the first face, and determining the first face shape feature and the first expression feature obtained from the last update as the target face shape feature and target expression feature of the first face, respectively, includes:

[0023] The first face shape feature, the first expression feature, and the first face rigid transformation matrix are used as weighting coefficients to fit the face vertex data of the multiple faces. The first face is used as the fitting target, and the first face shape feature, the first expression feature, and the first face rigid transformation matrix are iteratively updated to make the fitting result approach the first face. The first face shape feature and the first expression feature obtained in the last update are determined as the target face shape feature and target expression feature of the first face, respectively. The face rigid transformation matrix is ​​used to translate and rotate the face.

[0024] The target pose angle features of the first face are decomposed from the rigid transformation matrix of the face obtained from the last update;

[0025] The step of generating a target face image of the second face based on the target expression features of the first face, the target face shape features of the second face, and the target face texture features of the second face includes:

[0026] Based on the target expression features of the first face, the target pose angle features of the first face, the target face shape features of the second face, and the target face texture features of the second face, a target face image of the second face is generated.

[0027] In one possible implementation, the step of using the first face shape feature, the first expression feature, and the first face rigid transformation matrix as weighting coefficients to fit the face vertex data of the plurality of faces, and iteratively updating the first face shape feature, the first expression feature, and the first face rigid transformation matrix with the first face as the fitting target, so that the fitting result approximates the first face, includes:

[0028] Based on the preset first face shape features and the preset first expression features, the face vertex data of the multiple faces are fitted to obtain the first fitting result;

[0029] Based on the first fitting result and the face vertex data of the first face, the rigid transformation matrix of the first face is determined;

[0030] Based on the first face rigid transformation matrix, the first fitting result, and the face vertex data of the first face, the preset first face shape features and the preset first expression features are updated to obtain the updated first face shape features and the updated first expression features.

[0031] Based on the updated first face shape features and the updated first expression features, the face vertex data of the multiple faces are fitted to obtain a second fitting result;

[0032] Based on the second fitting result and the face vertex data of the first face, update the rigid transformation matrix of the first face;

[0033] Alternately update the rigid transformation matrix of the first face, the shape features of the first face, and the expression features of the first face until the iteration termination condition is met.

[0034] In one possible implementation, acquiring face vertex data includes:

[0035] Obtain the first set of face vertex data, where each set of face vertex data corresponds to one face. The first set of face vertex data is obtained by making the second set of expressions on the third set of faces.

[0036] The first set of face vertex data is dimensionality reduced to obtain a fourth set of face vertex data, wherein the fourth set is smaller than the first set.

[0037] In one possible implementation, the step of dimensionality reduction of the first set of face vertex data to obtain a fourth set of face vertex data includes:

[0038] Multiple sets of face vertex data that meet similar conditions in the first set of face vertex data are merged into one set of face vertex data to obtain the fourth set of face vertex data.

[0039] In one possible implementation, the face image of the first face is the face image of the first face in any video frame of the reference video, and the second face is the face of a virtual object; the method further includes:

[0040] For the face image of the first face in multiple video frames of the reference video, the steps of generating the target face image of the second face described above are executed sequentially to obtain the face images of the virtual face corresponding to the multiple video frames respectively;

[0041] Based on the order of the multiple video frames in the reference video, the face images of the virtual faces corresponding to the multiple video frames are displayed sequentially at the face position of the virtual object.

[0042] On the other hand, a face image generation apparatus is provided, the apparatus comprising:

[0043] The acquisition module is used to acquire the face image of the first face, the face image of the second face, and the face vertex data, wherein the face vertex data includes the face vertex data of multiple faces with different face shapes and different expressions;

[0044] The fitting module is used to fit the face vertex data of the plurality of faces using the first face shape feature and the first expression feature as weighting coefficients. Taking the first face as the fitting target, the first face shape feature and the first expression feature are iteratively updated so that the fitting result is close to the first face. The first face shape feature and the first expression feature obtained in the last update are respectively determined as the target face shape feature and the target expression feature of the first face.

[0045] The fitting module is further configured to use the second face shape feature and the second expression feature as weighting coefficients to fit the face vertex data of the plurality of faces, take the second face as the fitting target, iteratively update the second face shape feature and the second expression feature so that the fitting result is close to the second face, and determine the second face shape feature and the second expression feature obtained in the last update as the target face shape feature and the target expression feature of the second face, respectively.

[0046] The generation module is used to generate a target face image of the second face based on the target expression features of the first face, the target face shape features of the second face, and the target face texture features of the second face.

[0047] In one possible implementation, the fitting module includes:

[0048] The acquisition unit is used to acquire the face vertex data of the first face based on the face image of the first face;

[0049] The fitting unit is used to fit the face vertex data of the multiple faces by using the preset first face shape features and the preset first expression features as weighting coefficients.

[0050] The determining unit is used to determine the fitting error based on the fitting result and the face vertex data of the first face;

[0051] The update unit is used to iteratively update the preset first face shape features and the preset first expression features to reduce the fitting error.

[0052] In one possible implementation, the vertex data of the first face is two-dimensional data, and the vertex data of the plurality of faces is three-dimensional data; the determining unit is used to project the three-dimensional vertex data in the fitting result onto a two-dimensional coordinate system to obtain two-dimensional vertex data; the difference between the obtained two-dimensional vertex data and the vertex data of the first face is determined as the fitting error.

[0053] In one possible implementation, the face vertex data of the first face is the face key point data of the first face; the fitting unit is used to determine the face key point data of the plurality of faces that match the face key point data of the first face based on the face key point data of the first face; and to fit the face key point data of the plurality of faces using the preset first face shape feature and the preset first expression feature as weighting coefficients.

[0054] In one possible implementation, the fitting module is used to fit the face vertex data of the plurality of faces using the first face shape feature, the first expression feature, and the first face rigid transformation matrix as weighting coefficients. Taking the first face as the fitting target, the module iteratively updates the first face shape feature, the first expression feature, and the first face rigid transformation matrix to make the fitting result approximate the first face. The first face shape feature and the first expression feature obtained in the last update are determined as the target face shape feature and target expression feature of the first face, respectively. The face rigid transformation matrix is ​​used to translate and rotate the face. The target pose angle feature of the first face is decomposed from the face rigid transformation matrix obtained in the last update.

[0055] The generation module is used to generate a target face image of the second face based on the target expression features of the first face, the target pose angle features of the first face, the target face shape features of the second face, and the target face texture features of the second face.

[0056] In one possible implementation, the fitting module includes:

[0057] The fitting unit is used to fit the face vertex data of the multiple faces based on the preset first face shape features and the preset first expression features to obtain a first fitting result;

[0058] The determining unit is used to determine the first face rigid transformation matrix based on the first fitting result and the face vertex data of the first face;

[0059] The updating unit is used to update the preset first face shape features and the preset first expression features based on the first face rigid transformation matrix, the first fitting result and the face vertex data of the first face, so as to obtain the updated first face shape features and the updated first expression features.

[0060] The fitting unit is further configured to fit the face vertex data of the plurality of faces based on the updated first face shape features and the updated first expression features to obtain a second fitting result;

[0061] The updating unit is further configured to update the first face rigid transformation matrix based on the second fitting result and the face vertex data of the first face;

[0062] The update unit is also used to alternately update the rigid transformation matrix of the first face and the shape features and expression features of the first face until the iteration termination condition is met.

[0063] In one possible implementation, the acquisition module includes:

[0064] The acquisition unit is used to acquire a first number of sets of face vertex data, where each set of face vertex data corresponds to one face. The first number of sets of face vertex data are face vertex data obtained by making a second number of expressions on a third number of faces.

[0065] The dimensionality reduction unit is used to reduce the dimensionality of the first number of groups of face vertex data to obtain a fourth number of groups of face vertex data, wherein the fourth number is less than the first number.

[0066] In one possible implementation, the dimensionality reduction unit is used to fuse multiple sets of face vertex data that meet similar conditions in the first set of face vertex data into a single set of face vertex data to obtain the fourth set of face vertex data.

[0067] In one possible implementation, the face image of the first face is the face image of the first face in any video frame of the reference video, and the second face is the face of a virtual object; the apparatus further includes:

[0068] The fitting module and the generation module are used to sequentially execute the above-described steps of generating the target face image of the second face for the face image of the first face in multiple video frames of the reference video, so as to obtain the face images of the virtual face corresponding to the multiple video frames respectively.

[0069] The display module is used to sequentially display the face images of the virtual face corresponding to the multiple video frames at the face position of the virtual object, based on the order of the multiple video frames in the reference video.

[0070] On the other hand, a terminal is provided, the terminal including a processor and a memory, the memory storing at least one piece of program code, the at least one piece of program code being loaded and executed by the processor to perform the operations performed in the face image generation method as described above.

[0071] On the other hand, a computer-readable storage medium is provided that stores at least one piece of program code, which is loaded and executed by a processor to perform the operations performed in the face image generation method described above.

[0072] On the other hand, a computer program product is provided, which stores at least one piece of program code, which is loaded and executed by a processor to implement the operations performed in the face image generation method of the above embodiments.

[0073] This application provides a method, apparatus, terminal, and storage medium for generating face images. It can generate a second face image with the same expression as the first face based on a first face image and a second face image. Thus, it can generate a virtual face image with the same expression as a real face based on a real face image and a virtual face image. By making different expressions on a real face, the virtual face can be controlled to make the same expression, achieving virtual face control. This eliminates the need to build a virtual face model or collect motion data using expensive equipment, significantly reducing time and labor costs. Attached Figure Description

[0074] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0075] Figure 1 This is a schematic diagram of an implementation environment provided in an embodiment of this application;

[0076] Figure 2 This is a flowchart of a face image generation method provided in an embodiment of this application;

[0077] Figure 3This is a flowchart of a face image generation method provided in an embodiment of this application;

[0078] Figure 4 This is a flowchart of a face image generation method provided in an embodiment of this application;

[0079] Figure 5 This is a schematic diagram of the structure of a face image generation device provided in an embodiment of this application;

[0080] Figure 6 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application;

[0081] Figure 7 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0082] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0083] It is understood that the terms "first," "second," "third," "fourth," "fifth," "sixth," etc., used in this application may be used to describe various concepts herein, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of this application, the first face may be referred to as the second face, and the second face as the first face.

[0084] As used in this application, the terms "each," "multiple," "at least one," and "any" mean, with "at least one" including one, two, or more, "multiple" including two or more, "each" referring to each of the corresponding multiples, and "any" referring to any one of the multiples. For example, multiple faces include three faces, with "each" referring to every single one of the three faces, and "any" referring to any one of the three faces, which could be the first, the second, or the third.

[0085] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application are authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the face image of the first face, the face image of the second face, and the face vertex data in the embodiments of this application are authorized by the user or fully authorized by all parties, and the collection, use, and processing of the face image and face vertex data comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0086] This application embodiment can be applied to any scenario, such as video post-processing, short video production, live streaming, and virtual game scenarios. This application embodiment does not limit the application scenario. This application embodiment uses video post-processing as an example for illustrative explanation: In the post-production of science fiction movies, it is necessary to add dynamic CG (Computer Graphics) characters to the movie video. If the method provided in this application embodiment is used, a video of the user performing the facial movements required by the CG can be filmed first. Based on this video and the facial image of the CG character, a video of the CG character performing the facial movements can be generated and added to the movie video. This eliminates the need to build a 3D model of the CG character or use expensive equipment to collect motion data, greatly reducing time and labor costs.

[0087] The face image generation method provided in this application is executed by a computer device. In some embodiments, the computer device is a terminal, such as a mobile phone, tablet computer, laptop computer, desktop computer, etc., but is not limited thereto. In some embodiments, the computer device is a server, which may be a single server, a server cluster, or a cloud server providing services such as cloud computing and cloud storage. In some embodiments, the computer device includes a terminal and a server.

[0088] Figure 1 This is a schematic diagram of an implementation environment provided in an embodiment of this application. See also... Figure 1 The implementation environment includes terminal 101 and server 102. Terminal 101 and server 102 are connected via a wireless or wired network.

[0089] A target application, provided by server 102, is installed on terminal 101. Terminal 101 can perform functions such as data transmission and message interaction through this target application. Optionally, the target application is a target application in the operating system of terminal 101, or a target application provided by a third party. For example, the target application is a video processing application, a content sharing application, a game application, etc., and this application embodiment does not limit the target application. The target application has video processing functions; of course, the target application can also have other functions, such as review functions, shopping functions, navigation functions, etc. Optionally, server 102 is a backend server for the target application or a cloud server providing cloud computing and cloud storage services.

[0090] In some embodiments, terminal 101 sends a first face image and a second face image to server 102. Server 102 generates a target face image of the second face based on the first and second face images. The expression and facial pose angle of the target face image are the same as those of the first face. Of course, the above process can be completed jointly by terminal 101 and server 102. This application embodiment does not limit which steps terminal 101 and server 102 perform.

[0091] Figure 2 This is a flowchart illustrating a face image generation method provided in an embodiment of this application. This embodiment uses a terminal as the executing entity for illustrative purposes. See also... Figure 2 The method includes:

[0092] 201. The terminal acquires the face image of the first face, the face image of the second face, and the face vertex data, which includes the face vertex data of multiple faces with different face shapes and different expressions.

[0093] The first face can be the face of any object, such as a human face or a virtual object's face. The second face can also be the face of any object, such as a human face or a virtual object's face. This application does not limit the first and second faces; it only requires that the first and second faces are from different objects.

[0094] In this embodiment, the acquired face vertex data consists of face vertex data from multiple faces with different face shapes and expressions. The face vertex data for any given face consists of the coordinate data of multiple vertices on the face surface of that face, and these multiple vertices can be any point on the face surface of that face. In some embodiments, any face surface is divided into 10,000 grids, and one point is taken from each grid to obtain 10,000 points. The coordinate data of these 10,000 points are used as the face vertex data of that face.

[0095] 202. The terminal uses the first face shape feature and the first expression feature as weighting coefficients to fit the face vertex data of multiple faces. Taking the first face as the fitting target, the terminal iteratively updates the first face shape feature and the first expression feature to make the fitting result close to the first face. The first face shape feature and the first expression feature obtained in the last update are determined as the target face shape feature and target expression feature of the first face, respectively.

[0096] Because face vertex data includes vertex data from multiple faces with different shapes and expressions, it is possible to fit a face of any shape and expression based on this face vertex data. For example, if the face vertex data includes vertex data from a chubby face and a thin face, fitting the chubby face vertex data to the thin face vertex data can yield face vertex data for a face of medium chubby or thin face. Similarly, if the face vertex data includes vertex data from a laughing face and vertex data from a face with a natural expression, fitting the laughing face vertex data to the face with a natural expression can yield face vertex data for a smiling face.

[0097] In other words, by using an appropriate weighting coefficient to fit the vertex data of different faces, we can obtain the vertex data of a face with any shape; similarly, by using an appropriate weighting coefficient to fit the vertex data of faces with different expressions, we can obtain the vertex data of a face with any expression. Therefore, by using two weighting coefficients to fit the vertex data of multiple faces with different shapes and expressions, we can obtain the vertex data of a face with any shape and any expression. These two weighting coefficients can be used as the face shape features and expression features of the obtained face.

[0098] 203. The terminal uses the second face shape feature and the second expression feature as weighting coefficients to fit the face vertex data of multiple faces. Taking the second face as the fitting target, iteratively updates the second face shape feature and the second expression feature to make the fitting result approach the second face. The second face shape feature and the second expression feature obtained in the last update are determined as the target face shape feature and the target expression feature of the second face, respectively.

[0099] Step 203 is the same as step 202, and will not be described in detail here.

[0100] 204. The terminal generates a target face image of the second face based on the target expression features of the first face, the target face shape features of the second face, and the target face texture features of the second face.

[0101] The target face texture features are used to represent the skin tone, hair, and other texture information of the second face. The terminal generates a target face image of the second face based on the target expression features of the first face, the target face shape features of the second face, and the target face texture features of the second face. The expression of the second face in this target face image is the same as that of the first face. In other words, embodiments of this application can generate a face image of a second face with the same expression as the first face based on the face image of the first face and the face image of the second face.

[0102] For example, the face image of the first face represents the first face making a smiling expression, and the face image of the second face represents the second face with a natural expression. Based on the face images of the first face and the second face, a target face image representing the second face making a smiling expression can be generated.

[0103] The face image generation method provided in this application can generate a second face image with the same expression as the first face based on a first face image and a second face image. Thus, it can generate a virtual face image with the same expression as the real face based on a real face image and a virtual face image. By making different expressions on the real face, the virtual face can be controlled to make the same expression, realizing the driving of the virtual face. There is no need to build a virtual face model or collect motion data through expensive equipment, which greatly reduces time and labor costs.

[0104] Figure 3 This is a flowchart illustrating a face image generation method provided in an embodiment of this application. This embodiment uses a terminal as the executing entity for illustrative purposes; see [link to documentation]. Figure 3 The method includes:

[0105] 301. The terminal acquires the face image of the first face, the face image of the second face, and the face vertex data, which includes the face vertex data of multiple faces with different face shapes and expressions.

[0106] In some embodiments, the facial image of the first face can be obtained by capturing a photograph of the first face, or it can be stored locally, or it can be obtained from other devices. This application embodiment does not limit the acquisition of the facial image of the first face. In some embodiments, the facial image of the second face can be obtained by capturing a photograph of the second face, or it can be stored locally, or it can be obtained from other devices. This application embodiment does not limit the acquisition of the facial image of the second face.

[0107] In some embodiments, the first face is a real face, and the second face is a virtual face of a virtual object. The terminal can take a picture of the first face to obtain a face image of the first face, and the terminal can use a virtual face drawn by an artist as the face image of the second face.

[0108] In some embodiments, the face vertex data obtained in step 301 above consists of multiple sets of face vertex data, with one set of face vertex data corresponding to one face; the acquisition of face vertex data in step 301 includes: acquiring a first number of sets of face vertex data, where one set of face vertex data corresponds to one face, and the first number of sets of face vertex data is obtained by a third number of faces making a second number of expressions; and performing dimensionality reduction on the first number of sets of face vertex data to obtain a fourth number of sets of face vertex data, where the fourth number is less than the first number.

[0109] Optionally, the face vertex data is a 3D face kernel tensor database. The 3D face core tensor database is obtained by: each of n users making m facial expressions, acquiring the face vertex data of each of these n users when making these m facial expressions, and then reducing the dimensionality of the obtained m×n face vertex data to obtain z sets of face vertex data. In some embodiments, n is 1000, m is 52, and z is 50.

[0110] It should be noted that the terminal can use any statistical algorithm to reduce the dimensionality of the first set of face vertex data, and the specific method of dimensionality reduction is not limited in the embodiments of this application.

[0111] In some embodiments, the terminal performs dimensionality reduction on the first set of face vertex data to obtain a fourth set of face vertex data, including: fusing multiple sets of face vertex data that meet similar conditions in the first set of face vertex data into a single set of face vertex data to obtain the fourth set of face vertex data.

[0112] Optionally, satisfying the similarity condition means that the similarity between at least two sets of face vertex data reaches a target threshold. Optionally, the terminal clusters the first number of sets of face vertex data, and determines multiple sets of face vertex data belonging to the same cluster as multiple sets of face vertex data that satisfy the similarity condition; the embodiments of this application do not limit how to determine multiple sets of face vertex data that satisfy the similarity condition.

[0113] 302. The terminal uses the preset first face shape feature and the preset first expression feature as weighting coefficients to fit the face vertex data of multiple faces. Based on the fitting result and the face vertex data of the first face, the fitting error is determined, and the preset first face shape feature and the preset first expression feature are iteratively updated to reduce the fitting error.

[0114] Among them, the first face shape feature and the first expression feature can both be regarded as multi-dimensional vectors. The dimension of the vector is the same as the number of multiple faces. The value in one dimension is used to represent the weighting coefficient of the face vertex data of one face in the face vertex data of multiple faces.

[0115] The preset first face shape feature can be any face shape feature. Optionally, the preset first face shape feature is a default face shape feature; alternatively, the preset first face shape feature is a face shape feature set by a technician, etc. This application embodiment does not limit the preset first face shape feature. In some embodiments, the number of faces is z, and the preset first face shape feature is a z-dimensional vector, where the value in each dimension is 1 / z; the preset first face shape feature can be considered as the face shape feature of an average face.

[0116] The preset first expression feature can be any expression feature. Optionally, the preset first expression feature is a default expression feature; alternatively, the preset first expression feature is an expression feature set by a technician, etc. This application embodiment does not limit the preset first expression feature. In some embodiments, the number of multiple faces is z, the preset first expression feature is a z-dimensional vector, the value in the first dimension of the preset first expression feature is 1, and the values ​​in other dimensions are 0; the preset first expression feature can be regarded as the expression feature of a natural expression.

[0117] In some embodiments, when the terminal iteratively updates the preset first facial shape features and the preset first expression features, the updating stops when an iteration termination condition is met. Optionally, the iteration termination condition is that the number of iterations reaches a target number, which can be any number, such as 50 times, 60 times, etc., and this embodiment does not limit the target number. Optionally, the iteration termination condition is that the fitting error is less than an error threshold, which can be any value, and this embodiment does not limit the error threshold. Of course, the iteration termination condition can also be other conditions, and this embodiment does not limit the iteration termination condition.

[0118] In some embodiments, the vertex data of multiple faces are three-dimensional vertex data. The terminal uses the first face image as the target, and the vertex data obtained from the first face image is two-dimensional. Therefore, the terminal uses the two-dimensional vertex data of the first face as the target and fits the three-dimensional vertex data of multiple faces. Since the fitting result of fitting the vertex data of multiple faces is three-dimensional vertex data, the fitted three-dimensional vertex data can be mapped to a two-dimensional coordinate system. The mapped vertex data is then compared with the vertex data of the first face to determine whether the fitting result is similar to the first face.

[0119] In one possible implementation, the vertex data of the first face is two-dimensional data, while the vertex data of multiple faces is three-dimensional data. The terminal determines the fitting error based on the fitting result and the vertex data of the first face, including: projecting the three-dimensional vertex data from the fitting result onto a two-dimensional coordinate system to obtain two-dimensional vertex data; and determining the difference between the obtained two-dimensional vertex data and the vertex data of the first face as the fitting error. This fitting error can be called the reprojection error.

[0120] Optionally, the obtained two-dimensional face vertex data consists of the two-dimensional coordinates of multiple face vertices, and the face data of the first face consists of the two-dimensional coordinates of multiple face vertices in the first face. The difference between the obtained two-dimensional face vertex data and the face vertex data of the first face is the sum of the differences between the two-dimensional coordinates of each face vertex in the obtained two-dimensional face vertex data and the two-dimensional coordinates of the corresponding face vertex in the first face.

[0121] In some embodiments, the number of face vertices is large, resulting in a large amount of face vertex data. To reduce computational load, facial key points can be selected from the face vertices, and fitting can be performed solely based on these key points. Optionally, facial key points are points on the face contour and facial feature contour. In one possible implementation, the face vertex data of the first face is the facial key point data of the first face; the terminal uses preset first face shape features and preset first expression features as weighting coefficients to fit the face vertex data of the multiple faces, including: determining the facial key point data of multiple faces that match the facial key point data of the first face based on the facial key point data of the first face; and fitting the facial key point data of the multiple faces using preset first face shape features and preset first expression features as weighting coefficients.

[0122] In some embodiments, the terminal may employ a nonlinear optimization algorithm to iteratively update a preset first facial shape feature and a preset first expression feature. This nonlinear optimization algorithm can be any type of optimization algorithm, and the embodiments of this application do not limit the specific nonlinear optimization algorithm used. Optionally, the nonlinear optimization algorithm may be Newton's iteration method.

[0123] It should be noted that this embodiment only uses step 302 above as an example to illustrate the process of "using a first face as the target, iteratively updating the shape features and expression features of the first face, and using the updated shape features and expression features as weighting coefficients to fit the face vertex data of multiple faces." In another embodiment, the pose angles of the first face and the second face are different. For example, the first face is a side profile, and the second face is a frontal view. Or, the first face is looking down, and the second face is looking up. The terminal needs to generate a target face image for the second face with the same pose angle as the first face. Therefore, when fitting the face vertex data of multiple faces, the pose angle of the first face can also be referenced.

[0124] In one possible implementation, the face vertex data of multiple faces are fitted using the first face shape feature and the first expression feature as weighting coefficients. The first face is used as the fitting target, and the first face shape feature and the first expression feature are iteratively updated to make the fitting result approximate the first face. The first face shape feature and the first expression feature obtained from the last update are determined as the target face shape feature and target expression feature of the first face, respectively. This includes: using the first face shape feature, the first expression feature, and the first face rigid transformation matrix as weighting coefficients to fit the face vertex data of multiple faces; using the first face as the fitting target, the first face shape feature, the first expression feature, and the first face rigid transformation matrix are iteratively updated to make the fitting result approximate the first face; the first face shape feature and the first expression feature obtained from the last update are determined as the target face shape feature and target expression feature of the first face, respectively. The face rigid transformation matrix is ​​used to translate and rotate the face; the target pose angle feature of the first face is decomposed from the face rigid transformation matrix obtained from the last update.

[0125] Since the rigid transformation matrix of the face is used to translate and rotate the face, decomposing the rigid transformation matrix of the face can yield the rotation matrix of the face, that is, the target pose angle features of the first face.

[0126] In some embodiments, the first face shape feature, the first expression feature, and the first face rigid transformation matrix can be updated simultaneously. In some embodiments, the first face shape feature, the first expression feature, and the first face rigid transformation matrix can be updated alternately.

[0127] In one possible implementation, the first face shape feature, the first expression feature, and the first face rigid transformation matrix are updated alternately. That is, the first face rigid transformation matrix is ​​updated first, then the first face shape feature and the first expression feature are updated, then the first face rigid transformation matrix is ​​updated again, then the first face shape feature and the first expression feature are updated again, then the first face rigid transformation matrix is ​​updated again, and so on.

[0128] Optionally, taking the first face as the target, iteratively updating the first face shape features, the first expression features, and the first face rigid transformation matrix, and using the updated first face shape features, the updated first expression features, and the updated rigid face transformation matrix as weighting coefficients, fits the face vertex data of the multiple faces, including: fitting the face vertex data of the multiple faces based on preset first face shape features and preset first expression features to obtain a first fitting result; determining the first face rigid transformation matrix based on the first fitting result and the face vertex data of the first face; and determining the first face rigid transformation matrix based on the first face rigid transformation matrix and the first face face shape features, the updated first expression features, and the updated rigid face transformation matrix. The first face shape feature and the first face vertex data are used to update the preset first face shape feature and the preset first expression feature, resulting in updated first face shape feature and updated first expression feature. Based on the updated first face shape feature and the updated first expression feature, the face vertex data of multiple faces are fitted to obtain a second fitting result. Based on the second fitting result and the face vertex data of the first face, the first face rigid transformation matrix is ​​updated. The updates of the first face rigid transformation matrix and the first face shape feature and the first expression feature are alternately executed until the iteration termination condition is reached.

[0129] Alternatively, the fitting error can be expressed as:

[0130] ;

[0131] Here, argminf(x) represents the value of the variable when the objective function f(x) reaches its minimum value. Let represent the rigid transformation matrix of the first face, id represent the shape feature of the first face, and exp represent the expression feature of the first face. This represents the face vertex data of the i-th vertex in a face vertex data set of multiple faces; Let n be the vertex data of the i-th face of the first face, and n be the number of vertex pairs. Let x be the square of the 2-norm.

[0132] 303. The terminal determines the first face shape feature and the first expression feature obtained from the last update as the target face shape feature and target expression feature of the first face, respectively.

[0133] If the obtained fitting result is similar to the first face, it means that the facial shape features and expression features used can represent the facial shape and expression of the first face well. Therefore, the facial shape features and expression features used can be used as the target facial shape features and target expression features of the first face, that is, the real facial features and real expression features of the first face.

[0134] 304. The terminal uses the preset second face shape feature and the preset second expression feature as weighting coefficients to fit the face vertex data of multiple faces. Based on the fitting result and the face vertex data of the second face, the fitting error is determined, and the preset second face shape feature and the preset second expression feature are iteratively updated to reduce the fitting error.

[0135] 305. The terminal determines the second face shape feature and the second expression feature obtained from the last update as the target face shape feature and target expression feature of the second face, respectively.

[0136] It should be noted that steps 304 to 305 are similar to steps 302 to 303, and will not be described in detail here. Another point to note is that the preset second face shape feature is the same as the preset first face shape feature, and the preset second expression feature is the same as the preset first expression feature. Of course, the preset second face shape feature and the preset first face shape feature can also be different, and the preset second expression feature and the preset first expression feature can also be different. This application embodiment does not limit the preset first face shape feature, the preset second face shape feature, the preset first expression feature, and the preset second expression feature.

[0137] 306. The terminal acquires the target face texture features of the second face.

[0138] In some embodiments, the target facial texture features of the second face can be obtained locally or extracted by a feature extraction layer or a feature extraction model. This application does not limit the method of obtaining the target facial texture features.

[0139] In some embodiments, the target facial texture features include both shallow and deep features. The terminal acquires the facial texture features of a virtual face by: inputting a second face image into a plurality of connected feature extraction layers; sequentially extracting features from the second face image through the plurality of feature extraction layers to obtain the facial texture features output by each of the plurality of feature extraction layers; and determining the target facial texture features of the second face based on the facial texture features output by the plurality of feature extraction layers.

[0140] For example, the texture features of the target human face are:

[0141] ;

[0142] in, , and This represents the facial texture features output by the i-th feature extraction layer of the feature extraction model E.

[0143] 307. The terminal generates a target face image of the second face based on the target expression features of the first face, the target face shape features of the second face, and the target face texture features of the second face.

[0144] In some embodiments, the terminal may also acquire the target pose angle features of the first face and the second face. When generating the target face image of the second face, the terminal may also refer to the target pose angle features of the first face to ensure that the pose angle of the second face in the generated target face image is the same as that of the first face. The terminal generates the target face image of the second face based on the target expression features of the first face, the target face shape features of the second face, and the target face texture features of the second face, including: the terminal generates the target face image of the second face based on the target expression features of the first face, the target pose angle features of the first face, the target face shape features of the second face, and the target face texture features of the second face.

[0145] In some embodiments, the target face texture features include shallow texture features and deep texture features of the second face. In this way, when generating the target face image, the texture of the second face in the obtained target face image will not lose much information, and the obtained texture is similar to the original texture of the second face.

[0146] Optionally, the terminal generates a target face image of the second face based on the target expression features of the first face, the target face shape features of the second face, and the target face texture features of the second face. This includes: fusing the face texture features output by the first feature extraction layer with the target expression features of the first face and the target face shape features of the second face to obtain a first fused feature; fusing the face texture features output by any feature extraction layer after the first feature extraction layer with the fused feature corresponding to the face texture features output by the previous feature extraction layer, the face texture features output by the current feature extraction layer, the target expression features of the first face, and the target face shape features of the second face to obtain a second fused feature; and generating the target face image of the second face based on the fused feature corresponding to the face texture features output by the last feature extraction layer.

[0147] Optionally, a target face image of the second face is generated by a decoder based on the target expression features of the first face, the target face shape features of the second face, and the target face texture features of the second face. Optionally, the decoder is multi-layered, and the output of each layer can be:

[0148] ;

[0149] in, This represents the fused features of the decoder's layer i-th output. Let i represent the decoding function of the i-th layer, where... Depend on It is obtained by deconvolution. It represents the target facial expression features of the first face, the target facial shape features of the second face, and the target pose angle features of the second face.

[0150] In some embodiments, steps 302 to 307 are implemented by an image generation model, such as... Figure 4 As shown, the image generation model includes a 3D face feature encoder, a face texture feature encoder, and an expression-driven feature decoder. The face images of the first face and the second face are input into the 3D face feature encoder, which performs steps 302 to 305 to obtain the target pose angle features, target face shape features, and target expression features of the first face, as well as the target pose angle features, target face shape features, and target expression features of the second face. The face image of the second face is input into the face texture feature encoder, which performs step 306 to obtain the target face texture features of the second face. The target expression features and target pose angle features of the first face, and the face shape features and target face texture features of the second face are input into the expression-driven feature decoder, which performs step 307 to obtain the target face image of the second face.

[0151] The following provides an example of how to train the image generation model described above:

[0152] After inputting the face images of the first and second sample faces into the image generation model, the model outputs a predicted face image of the second sample face. Then, based on the differences in pose angle features between the predicted face image and the first sample face, the differences in expression features between the predicted face image and the first sample face, the differences in face shape features between the predicted face image and the second sample face, and the differences in face texture features between the predicted face image and the second sample face, the loss value of the image generation model is determined, and the model is trained based on this loss value.

[0153] For example, the loss value of this image generation model is:

[0154]

[0155] ;

[0156] ;

[0157] ;

[0158] ;

[0159] ;

[0160] Where L is the loss value, , These represent the face pose angle, face shape loss, expression loss, image reconstruction loss, and texture loss, respectively. and They represent and The corresponding loss weight. It is obtained by calculating the square of the 2-norm of the difference between the pose angle of the first sample face and the predicted face image. By calculating the facial shape features of the second sample faces Compared with the facial shape features of predicted face images The cosine similarity is obtained. By calculating and predicting facial expression features from human face images Facial expression features compared to the first sample face The square of the 2-norm of the difference is obtained. Predict face images by calculation Face images of the second sample face The 1-norm of the difference is obtained. Predicting face images using a face texture feature encoder. Face images of the second sample face The texture features are extracted layer by layer, and the 1-norm of the difference between the texture features of all layers is calculated.

[0161] In some embodiments, the face image of the first face is the face image of the first face in any video frame of the reference video, and the second face is the face of the virtual object; the method further includes: for the face images of the first face in multiple video frames of the reference video, sequentially performing the steps described above for generating the target face image of the second face, to obtain face images of the virtual face corresponding to each of the multiple video frames; based on the order of the multiple video frames in the reference video, sequentially displaying the face images of the virtual faces corresponding to each of the multiple video frames at the face position of the virtual object. In other words, by recording a video of the first face, the virtual face of the virtual object can be driven.

[0162] It should be noted that when generating virtual face images corresponding to multiple video frames based on the reference video, the target face texture features can be directly obtained from any device such as the terminal or server, or extracted by the feature extraction layer or feature extraction model. Alternatively, when generating the virtual face image corresponding to the first video frame, the target face texture features can be extracted by the feature extraction layer or feature extraction model, and the target face texture features can be used directly in subsequent iterations without needing to be extracted again.

[0163] The face image generation method provided in this application can generate a face image of a second face that makes the same expression as the first face based on a face image of a first face and a face image of a second face. That is, it can generate a face image of a virtual face that makes the same expression as the real face based on a face image of a real face and a face image of a virtual face. By making different expressions on the real face, the virtual face can be controlled to make the same expression, thus realizing the driving of the virtual face. There is no need to build a virtual face model or collect motion data through expensive equipment, which greatly reduces time and labor costs.

[0164] Figure 5 This is a schematic diagram of the structure of a face image generation device provided in an embodiment of this application. See also... Figure 5 The device includes:

[0165] The acquisition module 501 is used to acquire the face image of the first face, the face image of the second face, and the face vertex data, wherein the face vertex data includes the face vertex data of multiple faces with different face shapes and different expressions;

[0166] The fitting module 502 is used to fit the face vertex data of the plurality of faces using the first face shape feature and the first expression feature as weighting coefficients, taking the first face as the fitting target, iteratively updating the first face shape feature and the first expression feature so that the fitting result is close to the first face, and determining the first face shape feature and the first expression feature obtained in the last update as the target face shape feature and the target expression feature of the first face, respectively.

[0167] The fitting module 502 is further configured to use the second face shape feature and the second expression feature as weighting coefficients to fit the face vertex data of the plurality of faces, take the second face as the fitting target, iteratively update the second face shape feature and the second expression feature so that the fitting result is close to the second face, and determine the second face shape feature and the second expression feature obtained in the last update as the target face shape feature and the target expression feature of the second face, respectively.

[0168] The generation module 503 is used to generate a target face image of the second face based on the target expression features of the first face, the target face shape features of the second face, and the target face texture features of the second face.

[0169] In one possible implementation, the fitting module 502 includes:

[0170] The acquisition unit is used to acquire the face vertex data of the first face based on the face image of the first face;

[0171] The fitting unit is used to fit the face vertex data of the multiple faces by using the preset first face shape features and the preset first expression features as weighting coefficients.

[0172] The determining unit is used to determine the fitting error based on the fitting result and the face vertex data of the first face;

[0173] The update unit is used to iteratively update the preset first face shape features and the preset first expression features to reduce the fitting error.

[0174] In one possible implementation, the vertex data of the first face is two-dimensional data, and the vertex data of the plurality of faces is three-dimensional data; the determining unit is used to project the three-dimensional vertex data in the fitting result onto a two-dimensional coordinate system to obtain two-dimensional vertex data; the difference between the obtained two-dimensional vertex data and the vertex data of the first face is determined as the fitting error.

[0175] In one possible implementation, the face vertex data of the first face is the face key point data of the first face; the fitting unit is used to determine the face key point data of the plurality of faces that match the face key point data of the first face based on the face key point data of the first face; and to fit the face key point data of the plurality of faces using the preset first face shape feature and the preset first expression feature as weighting coefficients.

[0176] In one possible implementation, the fitting module 502 is used to fit the face vertex data of the plurality of faces using the first face shape feature, the first expression feature, and the first face rigid transformation matrix as weighting coefficients. Taking the first face as the fitting target, the module iteratively updates the first face shape feature, the first expression feature, and the first face rigid transformation matrix to make the fitting result approximate the first face. The first face shape feature and the first expression feature obtained in the last update are determined as the target face shape feature and target expression feature of the first face, respectively. The face rigid transformation matrix is ​​used to translate and rotate the face. The target pose angle feature of the first face is decomposed from the face rigid transformation matrix obtained in the last update.

[0177] The generation module 503 is used to generate a target face image of the second face based on the target expression features of the first face, the target pose angle features of the first face, the target face shape features of the second face, and the target face texture features of the second face.

[0178] In one possible implementation, the fitting module 502 includes:

[0179] The fitting unit is used to fit the face vertex data of the multiple faces based on the preset first face shape features and the preset first expression features to obtain a first fitting result;

[0180] The determining unit is used to determine the first face rigid transformation matrix based on the first fitting result and the face vertex data of the first face;

[0181] The updating unit is used to update the preset first face shape features and the preset first expression features based on the first face rigid transformation matrix, the first fitting result and the face vertex data of the first face, so as to obtain the updated first face shape features and the updated first expression features.

[0182] The fitting unit is further configured to fit the face vertex data of the plurality of faces based on the updated first face shape features and the updated first expression features to obtain a second fitting result;

[0183] The updating unit is further configured to update the first face rigid transformation matrix based on the second fitting result and the face vertex data of the first face;

[0184] The update unit is also used to alternately update the rigid transformation matrix of the first face and the shape features and expression features of the first face until the iteration termination condition is met.

[0185] In one possible implementation, the acquisition module 501 includes:

[0186] The acquisition unit is used to acquire a first number of sets of face vertex data, where each set of face vertex data corresponds to one face. The first number of sets of face vertex data are face vertex data obtained by making a second number of expressions on a third number of faces.

[0187] The dimensionality reduction unit is used to reduce the dimensionality of the first number of groups of face vertex data to obtain a fourth number of groups of face vertex data, wherein the fourth number is less than the first number.

[0188] In one possible implementation, the dimensionality reduction unit is used to fuse multiple sets of face vertex data that meet similar conditions in the first set of face vertex data into a single set of face vertex data to obtain the fourth set of face vertex data.

[0189] In one possible implementation, the face image of the first face is the face image of the first face in any video frame of the reference video, and the second face is the face of a virtual object; the apparatus further includes:

[0190] The fitting module 502 and the generation module 503 are used to sequentially execute the above-described steps of generating the target face image of the second face for the face image of the first face in multiple video frames of the reference video, so as to obtain the face images of the virtual face corresponding to the multiple video frames respectively.

[0191] The display module is used to sequentially display the face images of the virtual face corresponding to the multiple video frames at the face position of the virtual object, based on the order of the multiple video frames in the reference video.

[0192] Figure 6 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application. The terminal 600 can be a portable mobile terminal, such as a smartphone, tablet computer, laptop computer, desktop computer, smart home appliance, smartwatch, etc. The terminal 600 may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other names.

[0193] Terminal 600 includes a processor 601 and a memory 602.

[0194] Processor 601 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 601 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 601 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 601 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, processor 601 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0195] The memory 602 may include one or more computer-readable storage media, which may be non-transitory. The memory 602 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 602 are used to store at least one program code, which is executed by the processor 601 to implement the face image generation method provided in the method embodiments of this application.

[0196] In some embodiments, the terminal 600 may also optionally include a peripheral device interface 603 and at least one peripheral device. The processor 601, memory 602, and peripheral device interface 603 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 603 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 604, a display screen 605, a camera assembly 606, an audio circuit 607, a positioning assembly 608, and a power supply 609.

[0197] Peripheral interface 603 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 601 and memory 602. In some embodiments, processor 601, memory 602 and peripheral interface 603 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 601, memory 602 and peripheral interface 603 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0198] The radio frequency (RF) circuit 604 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 604 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 604 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 604 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 604 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 20G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 604 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0199] Display screen 605 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 605 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 601 for processing. In this case, display screen 605 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 605, disposed on the front panel of terminal 600; in other embodiments, there may be at least two display screens, disposed on different surfaces of terminal 600 or in a folded design; in other embodiments, display screen 605 may be a flexible display screen, disposed on a curved or folded surface of terminal 600. Furthermore, display screen 605 may be configured as a non-rectangular irregular shape, i.e., a non-rectangular screen. Display screen 605 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0200] The camera assembly 606 is used to acquire images or videos. Optionally, the camera assembly 606 includes a front-facing camera and a rear-facing camera. The front-facing camera is disposed on the front panel of the terminal, and the rear-facing camera is disposed on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 606 may also include a flash. The flash may be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cool light flash, which can be used for light compensation at different color temperatures.

[0201] The audio circuit 607 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 601 for processing, or input to the radio frequency circuit 604 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each located at a different part of the terminal 600. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert the electrical signals from the processor 601 or the radio frequency circuit 604 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 607 may also include a headphone jack.

[0202] The positioning component 608 is used to determine the current geographic location of the terminal 600 in order to enable navigation or LBS (Location Based Service). The positioning component 608 may be based on the US GPS (Global Positioning System), China's BeiDou system, Russia's Grenas positioning system, or the EU's Galileo positioning system.

[0203] Power supply 609 is used to supply power to the various components in terminal 600. Power supply 609 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 609 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, and a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0204] In some embodiments, the terminal 600 further includes one or more sensors 160. The one or more sensors 160 include, but are not limited to: an accelerometer 611, a gyroscope 612, a pressure sensor 613, a fingerprint sensor 614, an optical sensor 615, and a proximity sensor 616.

[0205] Accelerometer 611 can detect the magnitude of acceleration on the three coordinate axes of a coordinate system established with terminal 60. For example, accelerometer 611 can be used to detect the components of gravitational acceleration on the three coordinate axes. Processor 601 can control display screen 605 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 611. Accelerometer 611 can also be used for games or for acquiring user motion data.

[0206] The gyroscope sensor 612 can detect the orientation and rotation angle of the terminal 600. The gyroscope sensor 612, in conjunction with the accelerometer sensor 611, can collect the user's 3D movements on the terminal 600. Based on the data collected by the gyroscope sensor 612, the processor 601 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0207] The pressure sensor 613 can be disposed on the side bezel of the terminal 600 and / or on the lower layer of the display screen 605. When the pressure sensor 613 is disposed on the side bezel of the terminal 600, it can detect the user's grip signal on the terminal 600, and the processor 601 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 613. When the pressure sensor 613 is disposed on the lower layer of the display screen 605, the processor 601 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 605. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0208] The fingerprint sensor 614 is used to collect the user's fingerprint. The processor 601 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 614, or the fingerprint sensor 614 identifies the user's identity based on the collected fingerprint. When the user's identity is identified as trusted, the processor 601 authorizes the user to perform relevant sensitive operations, including unlocking the screen, viewing encrypted information, downloading software, making payments, and changing settings. The fingerprint sensor 614 can be located on the front, back, or side of the terminal 600. When the terminal 600 has physical buttons or a manufacturer's logo, the fingerprint sensor 614 can be integrated with the physical buttons or manufacturer's logo.

[0209] An optical sensor 615 is used to collect ambient light intensity. In one embodiment, the processor 601 can control the display brightness of the display screen 605 based on the ambient light intensity collected by the optical sensor 615. Specifically, when the ambient light intensity is high, the display brightness of the display screen 605 is increased; when the ambient light intensity is low, the display brightness of the display screen 605 is decreased. In another embodiment, the processor 601 can also dynamically adjust the shooting parameters of the camera assembly 606 based on the ambient light intensity collected by the optical sensor 615.

[0210] A proximity sensor 616, also known as a distance sensor, is installed on the front panel of the terminal 600. The proximity sensor 616 is used to detect the distance between the user and the front of the terminal 600. In one embodiment, when the proximity sensor 616 detects that the distance between the user and the front of the terminal 600 is gradually decreasing, the processor 601 controls the display screen 605 to switch from a screen-on state to a screen-off state; when the proximity sensor 616 detects that the distance between the user and the front of the terminal 600 is gradually increasing, the processor 601 controls the display screen 605 to switch from a screen-off state to a screen-on state.

[0211] Those skilled in the art will understand that Figure 6 The structure shown does not constitute a limitation on terminal 600, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0212] Figure 7 This is a schematic diagram of a server structure provided in an embodiment of this application. The server 700 can vary significantly due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 701 and one or more memories 702. The memory 702 stores at least one line of program code, which is loaded and executed by the processor 701 to implement the methods provided in the various method embodiments described above. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated upon here.

[0213] The server 700 is used to execute the steps performed by the server in the above method embodiments.

[0214] This application also provides a computer-readable storage medium storing at least one piece of program code, which is loaded and executed by a processor to implement the operations performed in the face image generation method of the above embodiments.

[0215] This application also provides a computer program product that stores at least one piece of program code, which is loaded and executed by a processor to implement the operations performed in the face image generation method of the above embodiments.

[0216] In some embodiments, the program code involved in the embodiments of this application may be deployed and executed on a server, or on multiple servers located in one location, or on multiple servers distributed in multiple locations and interconnected through a communication network. Multiple servers distributed in multiple locations and interconnected through a communication network may form a blockchain system.

[0217] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0218] The above are merely optional embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present application should be included within the protection scope of the present application.

Claims

1. A method for generating a human face image, characterized in that, The method includes: The process involves acquiring a first face image, a second face image, and face vertex data, wherein the face vertex data includes face vertex data of multiple faces with different face shapes and expressions. The first face shape feature and the first expression feature are used as weighting coefficients to fit the face vertex data of the multiple faces. The first face is used as the fitting target, and the first face shape feature and the first expression feature are iteratively updated so that the fitting result is close to the first face. The first face shape feature and the first expression feature obtained in the last update are determined as the target face shape feature and target expression feature of the first face, respectively. Using the second face shape feature and the second expression feature as weighting coefficients, the face vertex data of the multiple faces are fitted. Taking the second face as the fitting target, the second face shape feature and the second expression feature are iteratively updated so that the fitting result is close to the second face. The second face shape feature and the second expression feature obtained in the last update are determined as the target face shape feature and the target expression feature of the second face, respectively. Based on the target expression features of the first face, the target pose angle features of the first face, the target face shape features of the second face, and the target face texture features of the second face, a target face image of the second face is generated.

2. The method according to claim 1, characterized in that, The step of using the first face shape feature and the first expression feature as weighting coefficients to fit the face vertex data of the multiple faces, taking the first face as the fitting target, and iteratively updating the first face shape feature and the first expression feature to make the fitting result approximate the first face includes: Based on the face image of the first face, obtain the face vertex data of the first face; The preset first face shape feature and preset first expression feature are used as weighting coefficients to fit the face vertex data of the multiple faces; Based on the fitting results and the face vertex data of the first face, the fitting error is determined; The preset first face shape features and the preset first expression features are iteratively updated to reduce the fitting error.

3. The method according to claim 2, characterized in that, The vertex data of the first face is two-dimensional data, and the vertex data of the multiple faces are three-dimensional data; determining the fitting error based on the fitting result and the vertex data of the first face includes: The three-dimensional face vertex data in the fitting result is projected onto a two-dimensional coordinate system to obtain two-dimensional face vertex data; The difference between the obtained two-dimensional face vertex data and the face vertex data of the first face is determined as the fitting error.

4. The method according to claim 2, characterized in that, The vertex data of the first face is the key point data of the first face; The step of fitting the face vertex data of the multiple faces using preset first face shape features and preset first expression features as weighting coefficients includes: Based on the facial key point data of the first face, determine the facial key point data of the plurality of faces that match the facial key point data of the first face; The preset first face shape feature and the preset first expression feature are used as weighting coefficients to fit the facial key point data of the multiple faces.

5. The method according to claim 1, characterized in that, The step of using the first face shape feature and the first expression feature as weighting coefficients to fit the face vertex data of the plurality of faces, taking the first face as the fitting target, iteratively updating the first face shape feature and the first expression feature to make the fitting result approach the first face, and determining the first face shape feature and the first expression feature obtained from the last update as the target face shape feature and target expression feature of the first face, respectively, includes: The first face shape feature, the first expression feature, and the first face rigid transformation matrix are used as weighting coefficients to fit the face vertex data of the multiple faces. The first face is used as the fitting target, and the first face shape feature, the first expression feature, and the first face rigid transformation matrix are iteratively updated to make the fitting result approach the first face. The first face shape feature and the first expression feature obtained in the last update are determined as the target face shape feature and target expression feature of the first face, respectively. The face rigid transformation matrix is ​​used to translate and rotate the face. The target pose angle features of the first face are decomposed from the rigid transformation matrix of the face obtained from the last update.

6. The method according to claim 5, characterized in that, The step of using the first face shape feature, the first expression feature, and the first face rigid transformation matrix as weighting coefficients to fit the face vertex data of the plurality of faces, taking the first face as the fitting target, and iteratively updating the first face shape feature, the first expression feature, and the first face rigid transformation matrix to make the fitting result approximate the first face, includes: Based on the preset first face shape features and the preset first expression features, the face vertex data of the multiple faces are fitted to obtain the first fitting result; Based on the first fitting result and the face vertex data of the first face, the rigid transformation matrix of the first face is determined; Based on the first face rigid transformation matrix, the first fitting result, and the face vertex data of the first face, the preset first face shape features and the preset first expression features are updated to obtain the updated first face shape features and the updated first expression features. Based on the updated first face shape features and the updated first expression features, the face vertex data of the multiple faces are fitted to obtain a second fitting result; Based on the second fitting result and the face vertex data of the first face, update the rigid transformation matrix of the first face; Alternately update the rigid transformation matrix of the first face, the shape features of the first face, and the expression features of the first face until the iteration termination condition is met.

7. The method according to claim 1, characterized in that, The acquisition of face vertex data includes: Obtain the first set of face vertex data, where each set of face vertex data corresponds to one face. The first set of face vertex data is obtained by making the second set of expressions on the third set of faces. The first set of face vertex data is dimensionality reduced to obtain a fourth set of face vertex data, wherein the fourth set is smaller than the first set.

8. The method according to claim 7, characterized in that, The step of reducing the dimensionality of the first set of face vertex data to obtain the fourth set of face vertex data includes: Multiple sets of face vertex data that meet similar conditions in the first set of face vertex data are merged into one set of face vertex data to obtain the fourth set of face vertex data.

9. The method according to claim 1, characterized in that, The first face image is the face image of the first face in any video frame of the reference video, and the second face is the face of a virtual object; the method further includes: For the face image of the first face in multiple video frames of the reference video, the steps of generating the target face image of the second face described above are executed sequentially to obtain the face images of the virtual faces corresponding to the multiple video frames respectively; Based on the order of the multiple video frames in the reference video, the face images of the virtual faces corresponding to the multiple video frames are displayed sequentially at the face position of the virtual object.

10. A face image generation device, characterized in that, The device includes: The acquisition module is used to acquire the face image of the first face, the face image of the second face, and the face vertex data, wherein the face vertex data includes the face vertex data of multiple faces with different face shapes and different expressions; The fitting module is used to fit the face vertex data of the plurality of faces using the first face shape feature and the first expression feature as weighting coefficients. Taking the first face as the fitting target, the first face shape feature and the first expression feature are iteratively updated so that the fitting result is close to the first face. The first face shape feature and the first expression feature obtained in the last update are respectively determined as the target face shape feature and the target expression feature of the first face. The fitting module is further configured to use the second face shape feature and the second expression feature as weighting coefficients to fit the face vertex data of the plurality of faces, take the second face as the fitting target, iteratively update the second face shape feature and the second expression feature so that the fitting result is close to the second face, and determine the second face shape feature and the second expression feature obtained in the last update as the target face shape feature and the target expression feature of the second face, respectively. The generation module is used to generate a target face image of the second face based on the target expression features of the first face, the target pose angle features of the first face, the target face shape features of the second face, and the target face texture features of the second face.

11. A terminal, characterized in that, The terminal includes a processor and a memory, the memory storing at least one piece of program code, which is loaded and executed by the processor to perform the operations performed in the face image generation method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one piece of program code, which is loaded and executed by a processor to perform the operations performed in the face image generation method as described in any one of claims 1 to 9.

Images