Method for generating face images, terminal, and storage medium
Patent Information
- Application Number
- MYPI2025003119
- Authority / Receiving Office
- MY · MY
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2022-11-25
- Publication Date
- 2024-05-25
AI Technical Summary
Existing technology requires expensive equipment to collect fine motion data and build complex models when driving virtual objects' faces, resulting in high time and labor costs and unable to meet the need for low-cost and rapid iteration.
By obtaining the vertex data of multiple faces, the fitting algorithm is used to iteratively update the shape and expression features, generate the target face image, and realize the driving of the virtual face without the need to build a virtual face model or use expensive equipment.
It greatly reduces time and labor costs, realizes rapid driving of virtual faces, and meets the need for low-cost and rapid iteration.
Smart Images

Figure PI2025003119_DRAWING_001 
Figure PI2025003119_DRAWING_002 
Figure PI2025003119_DRAWING_003
Abstract
Description
Facial image generation method, device, equipment and storage medium Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus, device, and storage medium for generating a facial image. Background Art
[0002] With the continuous development of computer technology, virtual object driving has been widely used in many fields and has great market value. For example, in film production, virtual objects are added to the film and driven to complete the corresponding plot.
[0003] Currently, driving a virtual object requires building a three-dimensional model of the virtual object, and having professional action actors wear motion capture equipment and perform corresponding movements. The motion capture equipment records the motion data of each movement and maps the motion data to the three-dimensional model of the virtual object to drive the virtual object.
[0004] Since the facial driving of virtual objects requires a high degree of precision, it is necessary to obtain detailed facial motion data and build a detailed virtual object facial model. However, obtaining detailed facial motion data requires expensive motion capture equipment, and building a detailed virtual object facial model requires a long time and high manpower cost, which cannot meet the current demand for low cost and rapid iteration.
[0005] Summary of the Invention
[0006] The embodiments of the present application provide a method, apparatus, device, and storage medium for generating a facial image, which greatly reduces time and labor costs. The technical solution is as follows:
[0007] In one aspect, a method for generating a facial image is provided, the method comprising:
[0008] Acquire a facial image of a first face, acquire a facial image of a second face, and acquire facial vertex data, wherein the facial vertex data includes facial vertex data of multiple faces with different facial shapes and different expressions;
[0009] Using the first facial shape feature and the first expression feature as weighting coefficients, fitting the facial vertex data of the multiple faces, taking the first face as a fitting target, iteratively updating the first facial shape feature and the first expression feature so that the fitting result approaches the first face, and determining the first facial shape feature and the first expression feature obtained from the last update as the target facial shape feature and the target expression feature of the first face, respectively;
[0010] Using the second facial shape feature and the second facial expression feature as weighting coefficients, fitting the facial vertex data of the multiple faces, taking the second face as a fitting target, iteratively updating the second facial shape feature and the second facial expression feature so that the fitting result approaches the second face, and determining the second facial shape feature and the second facial expression feature obtained from the last update as the target facial shape feature and the target facial expression feature of the second face, respectively;
[0011] A target facial image of the second face is generated based on the target expression feature of the first face, the target facial shape feature of the second face, and the target facial texture feature of the second face.
[0012] In one possible implementation, using the first facial shape feature and the first facial expression feature as weighting coefficients, fitting the facial vertex data of the multiple faces, taking the first face as a fitting target, and iteratively updating the first facial shape feature and the first facial expression feature so that the fitting result approaches the first face, includes:
[0013] Based on the facial image of the first face, acquiring facial vertex data of the first face;
[0014] Using a preset first facial shape feature and a preset first expression feature as weighting coefficients, fitting facial vertex data of the multiple faces;
[0015] determining a fitting error based on the fitting result and the facial vertex data of the first face;
[0016] The preset first facial shape feature and the preset first expression feature are iteratively updated to reduce the fitting error.
[0017] In one possible implementation, the facial vertex data of the first face is two-dimensional data, and the facial vertex data of the multiple faces is three-dimensional data; and determining the fitting error based on the fitting result and the facial vertex data of the first face includes:
[0018] Projecting the three-dimensional facial vertex data in the fitting result into a two-dimensional coordinate system to obtain two-dimensional facial vertex data;
[0019] The difference between the obtained two-dimensional facial vertex data and the facial vertex data of the first face is determined as the fitting error.
[0020] In one possible implementation, the facial vertex data of the first face is facial key point data of the first face; and fitting the facial vertex data of the multiple faces using a preset first facial shape feature and a preset first expression feature as weighting coefficients includes:
[0021] Determining, based on the facial key point data of the first face, facial key point data of the multiple faces that match the facial key point data of the first face;
[0022] The preset first facial shape feature and the preset first expression feature are used as weighting coefficients to fit the facial key point data of the multiple faces.
[0023] In one possible implementation, the method includes using the first facial shape feature and the first facial expression feature as weighting coefficients to fit the facial vertex data of the multiple faces, taking the first face as a fitting target, iteratively updating the first facial shape feature and the first facial expression feature so that the fitting result approaches the first face, and determining the first facial shape feature and the first facial expression feature obtained from the last update as the target facial shape feature and the target facial expression feature of the first face, respectively, including:
[0024] Using the first facial shape feature, the first facial expression feature, and the first facial rigid transformation matrix as weighting coefficients, fitting facial vertex data of the multiple faces, taking the first face as a fitting target, iteratively updating the first facial shape feature, the first facial expression feature, and the first facial rigid transformation matrix so that the fitting result approaches the first face, and determining the first facial shape feature and the first facial expression feature obtained from the last update as the target facial shape feature and the target facial expression feature of the first face, respectively. The facial rigid transformation matrix is used to translate and rotate the face;
[0025] Decomposing the target pose angle feature of the first face from the face rigid transformation matrix obtained by the last update;
[0026] Generating a target facial image of the second face based on the target expression feature of the first face, the target facial shape feature of the second face, and the target facial texture feature of the second face includes:
[0027] A target facial image of the second face is generated based on the target expression feature of the first face, the target posture angle feature of the first face, the target facial shape feature of the second face, and the target facial texture feature of the second face.
[0028] In one possible implementation, using the first facial shape feature, the first facial expression feature, and the first facial rigid transformation matrix as weighting coefficients, fitting the facial vertex data of the multiple faces, taking the first face as a fitting target, and iteratively updating the first facial shape feature, the first facial expression feature, and the first facial rigid transformation matrix so that the fitting result approaches the first face, includes:
[0029] Fitting the facial vertex data of the multiple faces based on a preset first facial shape feature and a preset first expression feature to obtain a first fitting result;
[0030] Determining a first facial rigid transformation matrix based on the first fitting result and facial vertex data of the first face;
[0031] updating the preset first facial shape feature and the preset first expression feature based on the first facial rigid transformation matrix, the first fitting result, and the facial vertex data of the first face to obtain an updated first facial shape feature and an updated first expression feature;
[0032] Fitting the facial vertex data of the multiple faces based on the updated first facial shape feature and the updated first facial expression feature to obtain a second fitting result;
[0033] updating the first face rigid transformation matrix based on the second fitting result and the facial vertex data of the first face;
[0034] The first face rigid transformation matrix, the first face shape feature, and the first expression feature are updated alternately until an iteration termination condition is reached.
[0035] In a possible implementation, obtaining facial vertex data includes:
[0036] Obtaining a first number of groups of facial vertex data, where one group of facial vertex data corresponds to one face, and the first number of groups of facial vertex data is facial vertex data obtained when a third number of faces make a second number of expressions;
[0037] Perform dimensionality reduction on the first number of groups of facial vertex data to obtain a fourth number of groups of facial vertex data, where the fourth number is smaller than the first number.
[0038] In a possible implementation, performing dimensionality reduction on the first set of face vertex data to obtain a fourth set of face vertex data includes:
[0039] Multiple groups of facial vertex data that meet similar conditions in the first number of groups of facial vertex data are fused into one group of facial vertex data to obtain the fourth number of groups of facial vertex data.
[0040] In a possible implementation, the facial image of the first face is a facial image of the first face in any video frame in a reference video, and the second face is a face of a virtual object; the method further includes:
[0041] For the facial images of the first face in the multiple video frames of the reference video, sequentially performing the steps of generating the target facial image of the second face, to obtain facial images of the virtual face corresponding to the multiple video frames respectively;
[0042] Based on the order of the multiple video frames in the reference video, facial images of the virtual faces corresponding to the multiple video frames are displayed in sequence at the face position of the virtual object.
[0043] In another aspect, a facial image generation device is provided, the device comprising:
[0044] an acquisition module, configured to acquire a facial image of a first face, acquire a facial image of a second face, and acquire facial vertex data, wherein the facial vertex data includes facial vertex data of multiple faces with different facial shapes and different expressions;
[0045] a fitting module, configured to fit the facial vertex data of the multiple faces using the first facial shape feature and the first facial expression feature as weighting coefficients, taking the first face as a fitting target, iteratively updating the first facial shape feature and the first facial expression feature so that a fitting result approaches the first face, and determining the first facial shape feature and the first facial expression feature obtained from the last update as the target facial shape feature and the target facial expression feature of the first face, respectively;
[0046] The fitting module is further configured to use the second facial shape feature and the second facial expression feature as weighting coefficients to fit the facial vertex data of the multiple faces, take the second face as a fitting target, iteratively update the second facial shape feature and the second facial expression feature so that the fitting result approaches the second face, and determine the second facial shape feature and the second facial expression feature obtained from the last update as the target facial shape feature and the target facial expression feature of the second face, respectively;
[0047] A generating module is used to generate a target facial image of the second face based on the target expression feature of the first face, the target facial shape feature of the second face and the target facial texture feature of the second face.
[0048] In a possible implementation, the fitting module includes:
[0049] an acquiring unit, configured to acquire facial vertex data of the first face based on the facial image of the first face;
[0050] a fitting unit, configured to fit the facial vertex data of the plurality of faces by using a preset first facial shape feature and a preset first facial expression feature as weighting coefficients;
[0051] a determining unit, configured to determine a fitting error based on a fitting result and facial vertex data of the first face;
[0052] An updating unit is used to iteratively update the preset first facial shape feature and the preset first expression feature to reduce the fitting error.
[0053] In one possible implementation, the facial vertex data of the first face is two-dimensional data, and the facial vertex data of the multiple faces is three-dimensional data; the determination unit is used to project the three-dimensional facial vertex data in the fitting result into a two-dimensional coordinate system to obtain two-dimensional facial vertex data; and determine the difference between the obtained two-dimensional facial vertex data and the facial vertex data of the first face as the fitting error.
[0054] In one possible implementation, the facial vertex data of the first face is the facial key point data of the first face; the fitting unit is used to determine the facial key point data of the multiple faces that match the facial key point data of the first face based on the facial key point data of the first face; and fit the facial key point data of the multiple faces using the preset first facial shape feature and the preset first expression feature as weighting coefficients.
[0055] In one possible implementation, the fitting module is configured to use the first facial shape feature, the first facial expression feature, and the first facial rigid transformation matrix as weighting coefficients to fit facial vertex data of the multiple faces, take the first face as a fitting target, iteratively update the first facial shape feature, the first facial expression feature, and the first facial rigid transformation matrix so that the fitting result approaches the first face, determine the first facial shape feature and the first facial expression feature obtained in the last update as the target facial shape feature and the target facial expression feature of the first face, respectively, and use the facial rigid transformation matrix to translate and rotate the face; and decompose the target posture angle feature of the first face from the facial rigid transformation matrix obtained in the last update;
[0056] The generation module is used to generate a target facial image of the second face based on the target expression feature of the first face, the target posture angle feature of the first face, the target facial shape feature of the second face and the target facial texture feature of the second face.
[0057] In a possible implementation, the fitting module includes:
[0058] a fitting unit, configured to fit the facial vertex data of the plurality of faces based on a preset first facial shape feature and a preset first facial expression feature to obtain a first fitting result;
[0059] a determining unit, configured to determine a first face rigid transformation matrix based on the first fitting result and the facial vertex data of the first face;
[0060] an updating unit, configured to update the preset first facial shape feature and the preset first expression feature based on the first facial rigid transformation matrix, the first fitting result, and the facial vertex data of the first face, to obtain an updated first facial shape feature and an updated first expression feature;
[0061] The fitting unit is further configured to fit the facial vertex data of the multiple faces based on the updated first facial shape feature and the updated first facial expression feature to obtain a second fitting result;
[0062] The updating unit is further configured to update the first face rigid transformation matrix based on the second fitting result and the facial vertex data of the first face;
[0063] The updating unit is further configured to alternately update the first facial rigid transformation matrix, the first facial shape feature, and the first facial expression feature until an iteration termination condition is reached.
[0064] In a possible implementation, the acquisition module includes:
[0065] an acquiring unit, configured to acquire a first number of groups of facial vertex data, where each group of facial vertex data corresponds to a single face, and the first number of groups of facial vertex data are facial vertex data obtained when a third number of faces make a second number of expressions;
[0066] A dimensionality reduction unit is used to reduce the dimensionality of the first number of groups of facial vertex data to obtain a fourth number of groups of facial vertex data, where the fourth number is smaller than the first number.
[0067] In a possible implementation, the dimensionality reduction unit is configured to merge multiple groups of facial vertex data that meet similar conditions in the first number of groups of facial vertex data into one group of facial vertex data to obtain the fourth number of groups of facial vertex data.
[0068] In a possible implementation, the facial image of the first face is a facial image of the first face in any video frame in a reference video, and the second face is a face of a virtual object; the apparatus further includes:
[0069] The fitting module and the generating module are configured to sequentially perform the steps of generating the target facial image of the second face for the facial image of the first face in the plurality of video frames of the reference video, to obtain facial images of the virtual face corresponding to the plurality of video frames respectively;
[0070] The display module is configured to sequentially display facial images of the virtual faces corresponding to the multiple video frames at the face positions of the virtual objects based on the order of the multiple video frames in the reference video.
[0071] On the other hand, a terminal is provided, comprising a processor and a memory, wherein the memory stores at least one program code, and the at least one program code is loaded and executed by the processor to implement the operations performed in the facial image generation method described in the above aspects.
[0072] On the other hand, a computer-readable storage medium is provided, in which at least one program code is stored. The at least one program code is loaded and executed by a processor to implement the operations performed in the facial image generation method described in the above aspects.
[0073] On the other hand, a computer program product is provided, in which at least one program code is stored. The at least one program code is loaded and executed by a processor to implement the operations performed in the facial image generation method of the above embodiment.
[0074] The embodiments of the present application provide a facial image generation method, device, terminal and storage medium, which can generate a facial image of a second face making the same expression as the first face based on the facial image of a first face and the facial image of the second face, thereby generating a facial image of a virtual face making the same expression as the real face based on the facial image of a real face and the facial image of a virtual face. By making different expressions on the real face, the virtual face can be controlled to make the same expression, thereby realizing the driving of the virtual face. There is no need to build a model of the virtual face, nor to collect motion data through expensive equipment, which greatly reduces time and labor costs. BRIEF DESCRIPTION OF THE DRAWINGS
[0075] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0076] FIG1 is a schematic diagram of an implementation environment provided by an embodiment of the present application;
[0077] FIG2 is a flowchart of a method for generating a facial image provided by an embodiment of the present application;
[0078] FIG3 is a flowchart of a method for generating a facial image provided by an embodiment of the present application;
[0079] FIG4 is a flowchart of a method for generating a facial image provided by an embodiment of the present application;
[0080] FIG5 is a schematic structural diagram of a facial image generation device provided in an embodiment of the present application;
[0081] FIG6 is a schematic structural diagram of a terminal provided in an embodiment of the present application;
[0082] FIG7 is a schematic diagram of the structure of a server provided in an embodiment of the present application. DETAILED DESCRIPTION
[0083] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.
[0084] It is understood that the terms "first," "second," "third," "fourth," "fifth," "sixth," and so forth, as used herein, may be used to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are used solely to distinguish one concept from another. For example, a first face could be referred to as a second face, and a second face could be referred to as a first face, without departing from the scope of this application.
[0085] As used herein, the terms "each," "plurality," "at least one," and "any" include one, two, or more, "plurality" includes two or more, "each" refers to each of the corresponding plurality, and "any" refers to any one of the plurality. For example, if the plurality includes three faces, "each" refers to each of the three faces, and "any" refers to any one of the three faces, which can be the first, second, or third.
[0086] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the facial image of the first face, the facial image of the second face, and the facial vertex data in the embodiment of this application are authorized by the user or fully authorized by all parties, and the collection, use and processing of the facial image and facial vertex data comply with the relevant laws, regulations and standards of the relevant countries and regions.
[0087] The embodiments of the present application can be applied to any scenario such as video post-processing scenario, short video production scenario, live broadcast scenario, virtual game scenario, etc. The embodiments of the present application do not limit the application scenario. The embodiments of the present application are illustratively explained by taking the application in the video post-processing scenario as an example: in the post-production of science fiction movies, it is necessary to add dynamic CG (Computer Graphics) images to the movie video. If the method provided by the embodiments of the present application is adopted, a video of the user making the facial movements that the CG needs to make can be first shot, and based on the video and the facial image of the CG image, a video of the CG image making the facial movements is generated, and the video is added to the movie video. There is no need to build a three-dimensional model of the CG image, nor is there any need to use expensive equipment to collect motion data, which greatly reduces the time cost and labor cost.
[0088] The facial image generation method provided in the embodiments of the present application is performed by a computer device. In some embodiments, the computer device is a terminal, such as a mobile phone, tablet computer, laptop computer, desktop computer, etc., but is not limited thereto. In some embodiments, the computer device is a server, such as a single server, a server cluster, or a cloud server providing cloud computing and cloud storage services. In some embodiments, the computer device includes a terminal and a server.
[0089] Figure 1 is a schematic diagram of an implementation environment provided by an embodiment of the present application. Referring to Figure 1, the implementation environment includes a terminal 101 and a server 102. The terminal 101 and the server 102 are connected via a wireless or wired network.
[0090] The target application served by the server 102 is installed on the terminal 101, and the terminal 101 can realize functions such as data transmission and message interaction through the target application. Optionally, the target application is a target application in the operating system of the terminal 101, or a target application provided by a third party. For example, the target application is a video processing application, a content sharing application, a game application, etc., and the embodiment of the present application does not limit the target application. The target application has the function of video processing. Of course, the target application can also have other functions, such as review functions, shopping functions, navigation functions, etc. Optionally, the server 102 is the background server of the target application or a cloud server that provides cloud computing and cloud storage services.
[0091] In some embodiments, terminal 101 sends a facial image of a first face and a facial image of a second face to server 102. Server 102 generates a target facial image of the second face based on the facial image of the first face and the facial image of the second face, where the expression and facial posture angle of the target facial image are the same as those of the first face. Of course, the above process can be completed by terminal 101 and server 102 together, and the embodiments of the present application do not limit the steps performed by terminal 101 and server 102.
[0092] FIG2 is a flow chart of a facial image generation method provided in an embodiment of the present application. This embodiment of the present application is illustrated by taking a terminal as an example. Referring to FIG2 , the method includes:
[0093] 201. The terminal obtains a facial image of a first face, obtains a facial image of a second face, and obtains facial vertex data, where the facial vertex data includes facial vertex data of multiple faces with different facial shapes and different expressions.
[0094] The first face can be any face, such as a human face or a virtual face. The second face can also be any face, such as a human face or a virtual face. This embodiment of the application does not limit the first and second faces. It only requires that the first and second faces are different faces.
[0095] In the embodiments of the present application, the acquired facial vertex data is facial vertex data for multiple faces of varying facial shapes and expressions. The facial vertex data for any given face is the coordinate data for multiple vertices on the facial surface of that face, where each vertex can be any point on the facial surface of that face. In some embodiments, the facial surface of any given face is divided into 10,000 grids, and a point is taken from each grid, resulting in 10,000 points. The coordinate data for these 10,000 points is used as the facial vertex data for that face.
[0096] 202. The terminal uses the first facial shape feature and the first expression feature as weighting coefficients to fit the facial vertex data of multiple faces, takes the first face as the fitting target, iteratively updates the first facial shape feature and the first expression feature so that the fitting result approaches the first face, and determines the first facial shape feature and the first expression feature obtained from the last update as the target facial shape feature and target expression feature of the first face, respectively.
[0097] Because the facial vertex data includes the facial vertex data of multiple faces with different facial shapes and expressions, faces of any shape and expression can be fitted based on the facial vertex data. For example, if the facial vertex data includes the facial vertex data of a fat face and the facial vertex data of a thin face, the facial vertex data of the fat face can be fitted with the facial vertex data of the thin face to obtain the facial vertex data of a face with medium fatness or thinness. For another example, if the facial vertex data includes the facial vertex data of a laughing face and the facial vertex data of a face with a natural expression, the facial vertex data of the laughing face and the facial vertex data of the face with a natural expression can be fitted to obtain the facial vertex data of a smiling face.
[0098] In other words, by fitting the facial vertex data of different faces with a suitable weighting coefficient, facial vertex data for faces with any shape can be obtained; by fitting the facial vertex data of faces with different expressions with a suitable weighting coefficient, facial vertex data for faces with any expression can be obtained. Therefore, by fitting the facial vertex data of multiple faces with different faces and different expressions with two weighting coefficients, facial vertex data for faces with any shape and any expression can be obtained. These two weighting coefficients can be used as the facial shape features and expression features of the obtained faces.
[0099] 203. The terminal uses the second facial shape feature and the second expression feature as weighting coefficients to fit the facial vertex data of multiple faces, takes the second face as the fitting target, iteratively updates the second facial shape feature and the second expression feature so that the fitting result approaches the second face, and determines the second facial shape feature and the second expression feature obtained by the last update as the target facial shape feature and target expression feature of the second face, respectively.
[0100] The same process as step 202 is used in step 203 and will not be described in detail here.
[0101] 204. The terminal generates a target facial image of the second face based on the target expression feature of the first face, the target facial shape feature of the second face, and the target facial texture feature of the second face.
[0102] The target facial texture features are used to represent texture information such as skin color and hair of the second face. Based on the target expression features of the first face, the target facial shape features of the second face, and the target facial texture features of the second face, the terminal generates a target facial image of the second face, where the expression of the second face in the target facial image is the same as that of the first face. In other words, embodiments of the present application can generate a facial image of the second face with the same expression as the first face based on the facial image of the first face and the facial image of the second face.
[0103] For example, the facial image of the first face represents the first face making a smiling expression, and the facial image of the second face represents the second face with a natural expression. Based on the facial image of the first face and the facial image of the second face, a target facial image representing the second face with a smiling expression can be generated.
[0104] The facial image generation method provided in the embodiment of the present application can generate a facial image of a second face making the same expression as the first face based on the facial image of the first face and the facial image of the second face, thereby generating a facial image of a virtual face making the same expression as the real face based on the facial image of the real face and the facial image of the virtual face. By making different expressions on the real face, the virtual face can be controlled to make the same expression, thereby realizing the driving of the virtual face. There is no need to build a model of the virtual face, nor is there a need to collect motion data through expensive equipment, which greatly reduces time and labor costs.
[0105] FIG3 is a flow chart of a facial image generation method provided in an embodiment of the present application. The present application embodiment is exemplified by taking the execution subject as a terminal as an example. Referring to FIG3 , the method includes:
[0106] 301. The terminal obtains a facial image of a first face, obtains a facial image of a second face, and obtains facial vertex data, where the facial vertex data includes facial vertex data of multiple faces with different facial shapes and different expressions.
[0107] In some embodiments, the facial image of the first person may be obtained by photographing the first person, may be locally stored, or may be obtained from another device. The embodiments of the present application do not limit the manner in which the facial image of the first person is obtained. In some embodiments, the facial image of the second person may be obtained by photographing the second person, may be locally stored, or may be obtained from another device. The embodiments of the present application do not limit the manner in which the facial image of the second person is obtained.
[0108] In some embodiments, the first face is a real face, and the second face is a virtual face of a virtual object. The terminal can photograph the first face to obtain a facial image of the first face, and the terminal can use the virtual face drawn by the artist as the facial image of the second face.
[0109] In some embodiments, the facial vertex data obtained in the above step 301 is multiple groups of facial vertex data, and one group of facial vertex data corresponds to one face; obtaining the facial vertex data in the above step 301 includes: obtaining a first number of groups of facial vertex data, a group of facial vertex data corresponds to one face, and the first number of groups of facial vertex data are facial vertex data obtained by making a second number of expressions for a third number of faces; performing dimensionality reduction on the first number of groups of facial vertex data to obtain a fourth number of groups of facial vertex data, and the fourth number is smaller than the first number.
[0110] Optionally, the face vertex data is a three-dimensional face core tensor database F core The facial vertex data in the 3D facial core tensor database is obtained by: each of n users makes m expressions, obtaining facial vertex data for each of the n users making these m expressions, and performing dimensionality reduction on the obtained m×n facial vertex data to obtain z sets of facial vertex data. In some embodiments, n is 1000, m is 52, and z is 50.
[0111] It should be noted that the terminal can use any statistical algorithm to reduce the dimension of the first number of groups of face vertex data, and the embodiment of the present application does not limit the specific method of dimensionality reduction.
[0112] In some embodiments, the terminal performs dimensionality reduction on the first group of facial vertex data to obtain a fourth group of facial vertex data, including: fusing multiple groups of facial vertex data that meet similar conditions in the first group of facial vertex data into one group of facial vertex data to obtain the fourth group of facial vertex data.
[0113] Optionally, satisfying the similarity condition means that the similarity between at least two groups of facial vertex data reaches a target threshold. Optionally, the terminal clusters the first number of groups of facial vertex data and determines multiple groups of facial vertex data belonging to the same cluster as multiple groups of facial vertex data satisfying the similarity condition. This embodiment of the application does not limit how to determine the multiple groups of facial vertex data satisfying the similarity condition.
[0114] 302. The terminal uses the preset first facial shape feature and the preset first expression feature as weighting coefficients to fit the facial vertex data of multiple faces, determines the fitting error based on the fitting result and the facial vertex data of the first face, and iteratively updates the preset first facial shape feature and the preset first expression feature to reduce the fitting error.
[0115] Among them, the first facial shape feature and the first expression feature can be regarded as multidimensional vectors, the dimension of the vector is the same as the number of multiple faces, and the numerical value in one dimension is used to represent the weighted coefficient of the facial vertex data of one face among the facial vertex data of multiple faces.
[0116] The preset first facial shape feature can be any facial shape feature. Optionally, the preset first facial shape feature is a default facial shape feature; optionally, the preset first facial shape feature is a facial shape feature set by a technician, etc. The embodiments of the present application do not limit the preset first facial shape feature. In some embodiments, the number of multiple faces is z, and the preset first facial shape feature is a z-dimensional vector, with the value in each dimension being 1 / z; the preset first facial shape feature can be regarded as the facial shape feature of an average face.
[0117] The preset first expression feature can be any expression feature. Optionally, the preset first expression feature is a default expression feature; optionally, the preset first expression feature is an expression feature set by a technician, etc. The embodiment of the present application does not limit the preset first expression feature. In some embodiments, the number of multiple faces is z, the preset first expression feature is a z-dimensional vector, the value in the first dimension of the preset first expression feature is 1, and the values in other dimensions are 0; the preset first expression feature can be regarded as an expression feature of a natural expression.
[0118] In some embodiments, when the terminal iteratively updates the preset first facial shape feature and the preset first expression feature, the update is stopped when the iteration termination condition is reached. Optionally, the iteration termination condition is that the number of iterative updates reaches a target number of times, and the target number of times can be any number, for example, 50 times, 60 times, etc., and the embodiment of the present application does not limit the target number of times. Optionally, the iteration termination condition is that the fitting error is less than an error threshold, and the error threshold can be any value, and the embodiment of the present application does not limit the error threshold. Of course, the iteration termination condition can also be other conditions, and the embodiment of the present application does not limit the iteration termination condition.
[0119] In some embodiments, the facial vertex data of the multiple faces is three-dimensional facial vertex data. The terminal uses the first face as a target based on its facial image. The facial vertex data obtained from the facial image of the first face is two-dimensional facial vertex data. Therefore, the terminal uses the two-dimensional facial vertex data of the first face as a target to fit the three-dimensional facial vertex data of the multiple faces. Since the fitting result obtained after fitting the facial vertex data of the multiple faces is three-dimensional facial vertex data, the three-dimensional facial vertex data obtained by fitting can be mapped to a two-dimensional coordinate system. The mapped facial vertex data is then compared with the facial vertex data of the first face to determine whether the fitting result is similar to the first face.
[0120] In one possible implementation, the facial vertex data of the first face is two-dimensional data, and the facial vertex data of the multiple faces is three-dimensional data. The terminal determines a fitting error based on the fitting result and the facial vertex data of the first face, including: projecting the three-dimensional facial vertex data in the fitting result into a two-dimensional coordinate system to obtain two-dimensional facial vertex data; and determining the difference between the obtained two-dimensional facial vertex data and the facial vertex data of the first face as the fitting error. This fitting error can be referred to as a reprojection error.
[0121] Optionally, the obtained two-dimensional facial vertex data is the two-dimensional coordinates of multiple facial vertices, the facial data of the first face is the two-dimensional coordinates of multiple facial vertices in the first face, and the difference between the obtained two-dimensional facial vertex data and the facial vertex data of the first face is: the sum of the differences between the two-dimensional coordinates of each facial vertex in the obtained two-dimensional facial vertex data and the two-dimensional coordinates of the corresponding facial vertex in the first face.
[0122] In some embodiments, the number of facial vertices is relatively large, and the facial vertex data is relatively large. In order to reduce the amount of calculation, facial key points can be selected from the facial vertices, and fitting can be performed only based on the facial key points. Optionally, facial key points are points on the facial contour and facial features. In a possible implementation, the facial vertex data of the first face is the facial key point data of the first face; the terminal uses the preset first facial shape feature and the preset first expression feature as weighting coefficients to fit the facial vertex data of the multiple faces, including: based on the facial key point data of the first face, determining the facial key point data of multiple faces that match the facial key point data of the first face; using the preset first facial shape feature and the preset first expression feature as weighting coefficients to fit the facial key point data of multiple faces.
[0123] In some embodiments, the terminal may iteratively update the preset first facial shape feature and the preset first expression feature using a nonlinear optimization algorithm. The nonlinear optimization algorithm may be any optimization algorithm, and the embodiments of the present application do not limit the nonlinear optimization algorithm. Optionally, the nonlinear optimization algorithm is a Newton iteration method.
[0124] It should be noted that the embodiment of the present application only takes the above-mentioned step 302 as an example to exemplify "taking the first face as the target, iteratively updating the first facial shape feature and the first expression feature, using the updated first facial shape feature and the updated first expression feature as weighting coefficients, and fitting the facial vertex data of multiple faces." In another embodiment, the posture angles of the first face and the second face are different. For example, the first face is a side face, and the second face is a front face. For another example, the first face is lowering its head, and the second face is looking up, etc. The terminal needs to generate a target facial image for the second face with the same posture angle as the first face. Therefore, when fitting the facial vertex data of multiple faces, the posture angle of the first face can also be referred to.
[0125] In a possible implementation, the facial vertex data of multiple faces are fitted using the first facial shape feature and the first facial expression feature as weighting coefficients, the first face is taken as the fitting target, the first facial shape feature and the first facial expression feature are iteratively updated so that the fitting result approaches the first face, and the first facial shape feature and the first facial expression feature obtained from the last update are respectively determined as the target facial shape feature and the target facial expression feature of the first face, including: using the first facial shape feature, the first facial expression feature and the first facial rigid transformation matrix as weighting coefficients, the facial vertex data of multiple faces are fitted, the first face is taken as the fitting target, the first facial shape feature, the first facial expression feature and the first facial rigid transformation matrix are iteratively updated so that the fitting result approaches the first face, and the first facial shape feature and the first facial expression feature obtained from the last update are respectively determined as the target facial shape feature and the target expression feature of the first face, the facial rigid transformation matrix is used to translate and rotate the face; and the target posture angle feature of the first face is decomposed from the facial rigid transformation matrix obtained from the last update.
[0126] Since the face rigid transformation matrix is used to translate and rotate the face, the face rotation matrix can be obtained by decomposing the face rigid transformation matrix, that is, the target posture angle feature of the first face can be obtained.
[0127] In some embodiments, the first facial shape feature, the first expression feature, and the first facial rigid transformation matrix may be updated simultaneously. In some embodiments, the first facial shape feature, the first expression feature, and the first facial rigid transformation matrix may be updated alternately.
[0128] In one possible implementation, the first facial shape feature, the first expression feature and the first facial rigid transformation matrix are updated alternately, that is, the first facial rigid transformation matrix is updated first, then the first facial shape feature and the first expression feature are updated, then the first facial rigid transformation matrix is updated, then the first facial shape feature and the first expression feature are updated, then the first facial rigid transformation matrix is updated, and so on.
[0129] Optionally, the method takes the first face as the target, iteratively updates the first face shape feature, the first expression feature and the first face rigid transformation matrix, uses the updated first face shape feature, the updated first expression feature and the updated rigid face transformation matrix as weighting coefficients, and fits the face vertex data of the multiple faces, including: fitting the face vertex data of the multiple faces based on the preset first face shape feature and the preset first expression feature to obtain a first fitting result; determining the first face rigid transformation matrix based on the first fitting result and the face vertex data of the first face; and fitting the face vertex data of the multiple faces based on the first fitting result and the face vertex data of the first face. The method comprises the steps of: updating the preset first face shape feature and the preset first expression feature based on a fitting result and the facial vertex data of the first face to obtain an updated first face shape feature and an updated first expression feature; fitting the facial vertex data of the multiple faces based on the updated first face shape feature and the updated first expression feature to obtain a second fitting result; updating the first face rigid transformation matrix based on the second fitting result and the facial vertex data of the first face; and alternately performing the updating of the first face rigid transformation matrix, the first face shape feature, and the first expression feature until an iteration termination condition is reached.
[0130] Alternatively, the fitting error can be expressed as:
[0131]
[0132] Among them, argminf(x) represents the variable value when the objective function f(x) takes the minimum value. Rt represents the first face rigid transformation matrix, id represents the first face shape feature, exp represents the first expression feature, Represents the face vertex data of the i-th vertex in the face vertex data of multiple faces; s i is the vertex data of the i-th face vertex of the first face, n is the number of vertex pairs, Represents the square of the 2-norm of x.
[0133] 303. The terminal determines the first facial shape feature and the first expression feature obtained in the last update as the target facial shape feature and the target expression feature of the first face, respectively.
[0134] If the fitting result is similar to the first face, it means that the adopted facial shape features and expression features can better represent the facial shape and expression of the first face, so that the adopted facial shape features and expression features can be used as the target facial shape features and target expression features of the first face, that is, the real facial features and real expression features of the first face.
[0135] 304. The terminal uses the preset second facial shape feature and the preset second expression feature as weighting coefficients to fit the facial vertex data of multiple faces, determines the fitting error based on the fitting result and the facial vertex data of the second face, and iteratively updates the preset second facial shape feature and the preset second expression feature to reduce the fitting error.
[0136] 305. The terminal determines the second facial shape feature and the second expression feature obtained in the last update as the target facial shape feature and the target expression feature of the second face, respectively.
[0137] It should be noted that the above steps 304 to 305 are similar to the above steps 302 to 303, and will not be repeated here. Another point that needs to be explained is that the preset second facial shape feature is the same as the preset first facial shape feature, and the preset second expression feature is the same as the preset first expression feature. Of course, the preset second facial shape feature and the preset first facial shape feature may also be different, and the preset second expression feature and the preset first expression feature may also be different. The embodiment of the present application does not limit the preset first facial shape feature, the preset second facial shape feature, the preset first expression feature, and the preset second expression feature.
[0138] 306. The terminal obtains target facial texture features of the second face.
[0139] In some embodiments, the target facial texture features of the second face can be obtained locally, or can be extracted by a feature extraction layer or a feature extraction model. The embodiment of the present application does not limit the method for obtaining the target facial texture features.
[0140] In some embodiments, the target facial texture features include both shallow and deep features. The terminal obtains the facial texture features of the virtual face, including: inputting a facial image of a second face into a plurality of connected feature extraction layers; sequentially performing feature extraction on the facial image of the second face through the plurality of feature extraction layers to obtain facial texture features outputted by the plurality of feature extraction layers; and determining target facial texture features of the second face based on the facial texture features outputted by the plurality of feature extraction layers.
[0141] For example, the target face texture features are:
[0142]
[0143] Among them, f t is the target facial texture feature of the second face, and Represents the facial texture features output by the i-th feature extraction layer of the feature extraction model E.
[0144] 307. The terminal generates a target facial image of the second face based on the target expression feature of the first face, the target facial shape feature of the second face, and the target facial texture feature of the second face.
[0145] In some embodiments, the terminal may also obtain target posture angle features of the first face and target posture angle features of the second face. When generating a target facial image of the second face, the terminal may also refer to the target posture angle features of the first face so that the posture angle of the second face in the generated target facial image is the same as the posture angle of the first face. The terminal generates a target facial image of the second face based on the target expression features of the first face, the target facial shape features of the second face, and the target facial texture features of the second face, including: the terminal generates a target facial image of the second face based on the target expression features of the first face, the target posture angle features of the first face, the target facial shape features of the second face, and the target facial texture features of the second face.
[0146] In some embodiments, the target face texture features include shallow texture features and deep texture features of the second face. In this way, when generating the target face image, the texture of the second face in the target face image will not lose much information, and the obtained texture is similar to the original texture of the second face.
[0147] Optionally, the terminal generates a target facial image of the second face based on the target expression features of the first face, the target facial shape features of the second face and the target facial texture features of the second face, including: fusing the facial texture features output by the first feature extraction layer with the target expression features of the first face and the target facial shape features of the second face to obtain a first fusion feature; for the facial texture features output by any feature extraction layer after the first feature extraction layer, fusing the fusion features corresponding to the facial texture features output by the previous feature extraction layer, the facial texture features output by the feature extraction layer, the target expression features of the first face, and the target facial shape features of the second face to obtain a second fusion feature; and generating the target facial image of the second face based on the fusion features corresponding to the facial texture features output by the last feature extraction layer.
[0148] Optionally, a decoder is used to generate a target facial image of the second face based on the target expression features of the first face, the target facial shape features of the second face, and the target facial texture features of the second face. Optionally, the decoder is multi-layered, and the output of each layer can be:
[0149]
[0150] in, represents the fusion feature output of the i-th layer of the decoder, represents the decoding function of the i-th layer, where Depend on Calculated by deconvolution, f e Represents the target expression feature of the first face, the target facial shape feature of the second face, and the target posture angle feature of the second face.
[0151] In some embodiments, steps 302 to 307 are implemented by an image generation model, as shown in FIG4 , which includes a facial 3D feature encoder, a facial texture feature encoder, and an expression-driven feature decoder. The facial image of the first face and the facial image of the second face are input into the facial 3D feature encoder, which executes steps 302 to 305 to obtain target pose angle features, target facial shape features, and target expression features for the first face, as well as target pose angle features, target facial shape features, and target expression features for the second face; the facial image of the second face is input into the facial texture feature encoder, which executes step 306 to obtain target facial texture features for the second face; the target expression features and target pose angle features for the first face, and the facial shape features and target facial texture features for the second face are input into the expression-driven feature decoder, which executes step 307 to obtain the target facial image for the second face.
[0152] The following is an example of how to train the above image generation model:
[0153] After inputting a facial image of a first sample face and a facial image of a second sample face into an image generation model, the image generation model outputs a predicted facial image of the second sample face. Subsequently, a loss value of the image generation model is determined based on differences between the pose angle features of the predicted facial image and the pose angle features of the first sample face, differences between the expression features of the predicted facial image and the pose angle features of the first sample face, differences between the facial shape features of the predicted facial image and the facial shape features of the second sample face, and differences between the facial texture features of the predicted facial image and the facial texture features of the second sample face, and the image generation model is trained based on the loss value.
[0154] For example, the loss value of this image generation model is:
[0155] L=λ θ L θ +λ id L id +λ exp L exp +λ rec L rec +λ tex L tex ;
[0156]
[0157]
[0158]
[0159] L rec =‖I d -I a ‖1;
[0160]
[0161] Among them, L is the loss value, L θ , L id , L exp , L rec and L tex They represent face pose angle, face shape loss, expression loss, image reconstruction loss and texture loss respectively, and λ θ ,λ id ,λ exp ,λ rec and λ tex Respectively represent L θ , L id , L exp , L rec and L tex The corresponding loss weight. L θ It is obtained by calculating the square of the 2-norm of the difference between the pose angle of the first sample face and the predicted face image. id By calculating the facial shape feature id of the second sample face d and the facial shape feature id of the predicted face image a The cosine similarity is obtained, L exp Predict the facial expression features exp by calculating d The expression feature exp of the first sample face r The square of the 2-norm of the difference is obtained, L rec Predicting face image I by calculation d The face image I of the second sample face a The 1-norm of the difference is obtained. L tex Predicting face image I through face texture feature encoder d The face image I of the second sample face a The texture features are extracted layer by layer, and the 1-norm of the texture feature difference of all layers is calculated.
[0162] In some embodiments, the facial image of the first face is a facial image of the first face in any video frame of a reference video, and the second face is a face of a virtual object. The method further includes: sequentially performing the above-mentioned steps of generating a target facial image of the second face for the facial images of the first face in multiple video frames of the reference video, to obtain facial images of the virtual faces corresponding to the multiple video frames; and sequentially displaying the facial images of the virtual faces corresponding to the multiple video frames at the face position of the virtual object based on the order of the multiple video frames in the reference video. In other words, the virtual face of the virtual object can be driven by recording a video of the first face.
[0163] It should be noted that when generating facial images of virtual faces corresponding to multiple video frames based on a reference video, the target facial texture features can be directly obtained from any device such as a terminal or server, or can be extracted by a feature extraction layer or a feature extraction model. The target facial texture features can also be extracted by the feature extraction layer or the feature extraction model when generating the facial image of the virtual face corresponding to the first video frame, and the target facial texture features can be directly used subsequently without the need for re-extraction.
[0164] The facial image generation method provided in the embodiment of the present application can generate a facial image of a second face making the same expression as the first face based on the facial image of the first face and the facial image of the second face. That is, based on the facial image of the real face and the facial image of the virtual face, a facial image of a virtual face making the same expression as the real face can be generated. By making different expressions of the real face, the virtual face can be controlled to make the same expression, thereby realizing the driving of the virtual face. There is no need to build a model of the virtual face, nor is there a need to collect motion data through expensive equipment, which greatly reduces time and labor costs.
[0165] FIG5 is a schematic diagram of the structure of a facial image generation device provided in an embodiment of the present application. Referring to FIG5 , the device includes:
[0166] An acquisition module 501 is configured to acquire a facial image of a first face, acquire a facial image of a second face, and acquire facial vertex data, wherein the facial vertex data includes facial vertex data of multiple faces with different facial shapes and expressions;
[0167] A fitting module 502 is configured to fit the facial vertex data of the multiple faces using the first facial shape feature and the first facial expression feature as weighting coefficients, taking the first face as a fitting target, iteratively updating the first facial shape feature and the first facial expression feature so that the fitting result approaches the first face, and determining the first facial shape feature and the first facial expression feature obtained from the last update as the target facial shape feature and the target facial expression feature, respectively, for the first face;
[0168] The fitting module 502 is further configured to use the second facial shape feature and the second facial expression feature as weighting coefficients to fit the facial vertex data of the multiple faces, take the second face as a fitting target, iteratively update the second facial shape feature and the second facial expression feature so that the fitting result approaches the second face, and determine the second facial shape feature and the second facial expression feature obtained from the last update as the target facial shape feature and the target facial expression feature of the second face, respectively.
[0169] The generating module 503 is configured to generate a target facial image of the second face based on the target expression feature of the first face, the target facial shape feature of the second face, and the target facial texture feature of the second face.
[0170] In a possible implementation, the fitting module 502 includes:
[0171] an acquiring unit, configured to acquire facial vertex data of the first face based on the facial image of the first face;
[0172] a fitting unit, configured to fit the facial vertex data of the plurality of faces by using a preset first facial shape feature and a preset first facial expression feature as weighting coefficients;
[0173] a determining unit, configured to determine a fitting error based on a fitting result and facial vertex data of the first face;
[0174] An updating unit is used to iteratively update the preset first facial shape feature and the preset first expression feature to reduce the fitting error.
[0175] In one possible implementation, the facial vertex data of the first face is two-dimensional data, and the facial vertex data of the multiple faces is three-dimensional data; the determination unit is used to project the three-dimensional facial vertex data in the fitting result into a two-dimensional coordinate system to obtain two-dimensional facial vertex data; and determine the difference between the obtained two-dimensional facial vertex data and the facial vertex data of the first face as the fitting error.
[0176] In one possible implementation, the facial vertex data of the first face is the facial key point data of the first face; the fitting unit is used to determine the facial key point data of the multiple faces that match the facial key point data of the first face based on the facial key point data of the first face; and fit the facial key point data of the multiple faces using the preset first facial shape feature and the preset first expression feature as weighting coefficients.
[0177] In one possible implementation, the fitting module 502 is configured to use the first facial shape feature, the first facial expression feature, and the first facial rigid transformation matrix as weighting coefficients to fit facial vertex data of the multiple faces, take the first face as a fitting target, iteratively update the first facial shape feature, the first facial expression feature, and the first facial rigid transformation matrix so that the fitting result approaches the first face, determine the first facial shape feature and the first facial expression feature obtained in the last update as the target facial shape feature and the target facial expression feature of the first face, respectively, and use the facial rigid transformation matrix to translate and rotate the face; and decompose the target posture angle feature of the first face from the facial rigid transformation matrix obtained in the last update.
[0178] The generating module 503 is configured to generate a target facial image of the second face based on the target expression feature of the first face, the target posture angle feature of the first face, the target facial shape feature of the second face, and the target facial texture feature of the second face.
[0179] In a possible implementation, the fitting module 502 includes:
[0180] a fitting unit, configured to fit the facial vertex data of the plurality of faces based on a preset first facial shape feature and a preset first facial expression feature to obtain a first fitting result;
[0181] a determining unit, configured to determine a first face rigid transformation matrix based on the first fitting result and the facial vertex data of the first face;
[0182] an updating unit, configured to update the preset first facial shape feature and the preset first expression feature based on the first facial rigid transformation matrix, the first fitting result, and the facial vertex data of the first face, to obtain an updated first facial shape feature and an updated first expression feature;
[0183] The fitting unit is further configured to fit the facial vertex data of the multiple faces based on the updated first facial shape feature and the updated first facial expression feature to obtain a second fitting result;
[0184] The updating unit is further configured to update the first face rigid transformation matrix based on the second fitting result and the facial vertex data of the first face;
[0185] The updating unit is further configured to alternately update the first facial rigid transformation matrix, the first facial shape feature, and the first facial expression feature until an iteration termination condition is reached.
[0186] In a possible implementation, the acquisition module 501 includes:
[0187] an acquiring unit, configured to acquire a first number of groups of facial vertex data, where each group of facial vertex data corresponds to a single face, and the first number of groups of facial vertex data are facial vertex data obtained when a third number of faces make a second number of expressions;
[0188] A dimensionality reduction unit is used to reduce the dimensionality of the first number of groups of facial vertex data to obtain a fourth number of groups of facial vertex data, where the fourth number is smaller than the first number.
[0189] In a possible implementation, the dimensionality reduction unit is configured to merge multiple groups of facial vertex data that meet similar conditions in the first number of groups of facial vertex data into one group of facial vertex data to obtain the fourth number of groups of facial vertex data.
[0190] In a possible implementation, the facial image of the first face is a facial image of the first face in any video frame in a reference video, and the second face is a face of a virtual object; the apparatus further includes:
[0191] The fitting module 502 and the generating module 503 are configured to sequentially perform the steps of generating the target facial image of the second face for the facial image of the first face in the multiple video frames of the reference video, to obtain facial images of the virtual face corresponding to the multiple video frames respectively;
[0192] The display module is configured to sequentially display facial images of the virtual faces corresponding to the multiple video frames at the face positions of the virtual objects based on the order of the multiple video frames in the reference video.
[0193] Figure 6 is a schematic diagram of the structure of a terminal provided in an embodiment of the present application. Terminal 600 may be a portable mobile terminal, such as a smartphone, tablet computer, laptop computer, desktop computer, smart home appliance, smartwatch, etc. Terminal 600 may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other names.
[0194] The terminal 600 includes a processor 601 and a memory 602 .
[0195] The processor 601 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 601 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 601 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 601 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 601 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.
[0196] The memory 602 may include one or more computer-readable storage media, which may be non-transitory. The memory 602 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 602 is used to store at least one program code, which is executed by the processor 601 to implement the facial image generation method provided in the method embodiment of the present application.
[0197] In some embodiments, terminal 600 may optionally include a peripheral device interface 603 and at least one peripheral device. Processor 601, memory 602, and peripheral device interface 603 may be connected via a bus or signal lines. Each peripheral device may be connected to peripheral device interface 603 via a bus, signal lines, or circuit boards. Specifically, the peripheral device may include at least one of a radio frequency circuit 604, a display screen 605, a camera assembly 606, an audio circuit 607, a positioning assembly 608, and a power supply 609.
[0198] The peripheral device interface 603 can be used to connect at least one I / O (Input / Output)-related peripheral device to the processor 601 and the memory 602. In some embodiments, the processor 601, the memory 602, and the peripheral device interface 603 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 601, the memory 602, and the peripheral device interface 603 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.
[0199] The radio frequency circuit 604 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 604 communicates with communication networks and other communication devices via electromagnetic signals. The radio frequency circuit 604 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 604 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 604 can communicate with other terminals via at least one wireless communication protocol. Such wireless communication protocols include, but are not limited to, the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 20G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the radio frequency circuit 604 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.
[0200] Display screen 605 is used to display a user interface (UI). This UI may include graphics, text, icons, videos, or any combination thereof. When display screen 605 is a touchscreen display, it is also capable of collecting touch signals on or above the surface of display screen 605. These touch signals can be input as control signals to processor 601 for processing. Display screen 605 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, display screen 605 may be a single display screen, located on the front panel of terminal 600. In other embodiments, display screen 605 may be at least two, located on different surfaces of terminal 600 or in a foldable design. In still other embodiments, display screen 605 may be a flexible display screen, located on a curved or foldable surface of terminal 600. Display screen 605 can also be configured as a non-rectangular, irregular shape, also known as a special-shaped screen. Display screen 605 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).
[0201] The camera assembly 606 is used to capture images or videos. Optionally, the camera assembly 606 includes a front camera and a rear camera. The front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 606 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. The dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation under different color temperatures.
[0202] The audio circuit 607 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals that are input into the processor 601 for processing, or input into the radio frequency circuit 604 to achieve voice communication. For the purpose of stereo sound collection or noise reduction, there may be multiple microphones, each located in different parts of the terminal 600. The microphone may also be an array microphone or an omnidirectional collection microphone. The speaker is used to convert electrical signals from the processor 601 or the radio frequency circuit 604 into sound waves. The speaker may be a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert electrical signals into sound waves audible to humans, but also convert electrical signals into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 607 may also include a headphone jack.
[0203] Positioning component 608 is used to locate the current geographic location of terminal 600 to implement navigation or LBS (Location Based Service). Positioning component 608 can be based on the US GPS (Global Positioning System), China's Beidou system, Russia's Greninja positioning system, or the European Union's Galileo positioning system.
[0204] Power supply 609 is used to power various components in terminal 600. Power supply 609 can be AC power, DC power, a disposable battery, or a rechargeable battery. When power supply 609 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.
[0205] In some embodiments, the terminal 600 further includes one or more sensors 610 , including but not limited to: an acceleration sensor 611 , a gyroscope sensor 612 , a pressure sensor 613 , a fingerprint sensor 614 , an optical sensor 615 , and a proximity sensor 616 .
[0206] The accelerometer 611 can detect the magnitude of acceleration along the three coordinate axes of the coordinate system established by the terminal 60. For example, the accelerometer 611 can be used to detect the components of gravity acceleration along the three coordinate axes. The processor 601 can control the display screen 605 to display the user interface in a landscape or portrait view based on the gravity acceleration signal collected by the accelerometer 611. The accelerometer 611 can also be used to collect game or user motion data.
[0207] The gyroscope sensor 612 can detect the orientation and rotation angle of the terminal 600. It can work with the accelerometer 611 to collect the user's 3D movements on the terminal 600. Based on the data collected by the gyroscope sensor 612, the processor 601 can implement the following functions: motion sensing (for example, changing the UI based on the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.
[0208] The pressure sensor 613 can be set on the side frame of the terminal 600 and / or the lower layer of the display screen 602. When the pressure sensor 613 is set on the side frame of the terminal 600, it can detect the user's grip signal of the terminal 600, and the processor 601 performs left and right hand recognition or shortcut operations based on the grip signal collected by the pressure sensor 613. When the pressure sensor 613 is set on the lower layer of the display screen 602, the processor 601 controls the operable controls on the UI interface based on the user's pressure operation on the display screen 605. The operable controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.
[0209] The fingerprint sensor 614 is used to collect the user's fingerprint. The processor 601 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 614, or the fingerprint sensor 614 identifies the user's identity based on the collected fingerprint. When the user's identity is recognized as a trusted identity, the processor 601 authorizes the user to perform relevant sensitive operations, such as unlocking the screen, viewing encrypted information, downloading software, making payments, and changing settings. The fingerprint sensor 614 can be set on the front, back, or side of the terminal 600. When a physical button or manufacturer logo is provided on the terminal 600, the fingerprint sensor 614 can be integrated with the physical button or manufacturer logo.
[0210] Optical sensor 615 is used to collect ambient light intensity. In one embodiment, processor 601 can control the display brightness of display screen 605 based on the ambient light intensity collected by optical sensor 615. Specifically, when the ambient light intensity is high, the display brightness of display screen 605 is increased; when the ambient light intensity is low, the display brightness of display screen 605 is decreased. In another embodiment, processor 601 can also dynamically adjust the shooting parameters of camera assembly 606 based on the ambient light intensity collected by optical sensor 615.
[0211] Proximity sensor 616, also known as a distance sensor, is provided on the front panel of terminal 600. Proximity sensor 616 is used to detect the distance between the user and the front of terminal 600. In one embodiment, when proximity sensor 616 detects that the distance between the user and the front of terminal 600 is gradually decreasing, processor 601 controls display screen 605 to switch from the screen-on state to the screen-off state. When proximity sensor 616 detects that the distance between the user and the front of terminal 600 is gradually increasing, processor 601 controls display screen 605 to switch from the screen-off state to the screen-on state.
[0212] Those skilled in the art will appreciate that the structure shown in FIG6 does not limit the terminal 600 and may include more or fewer components than shown, or combine certain components, or adopt a different component arrangement.
[0213] FIG7 is a schematic diagram of the structure of a server provided in an embodiment of the present application. The server 700 may vary significantly due to different configurations or performances, and may include one or more processors (Central Processing Units, CPUs) 701 and one or more memories 702. The memories 702 store at least one program code, which is loaded and executed by the processor 701 to implement the methods provided in the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input / output interface for input and output. The server may also include other components for implementing device functions, which will not be described in detail here.
[0214] The server 700 is used to execute the steps executed by the server in the above method embodiment.
[0215] An embodiment of the present application also provides a computer-readable storage medium, which stores at least one program code. The at least one program code is loaded and executed by a processor to implement the operations performed in the facial image generation method of the above embodiment.
[0216] An embodiment of the present application also provides a computer program product, which stores at least one program code, and the at least one program code is loaded and executed by a processor to implement the operations performed in the facial image generation method of the above embodiment.
[0217] In some embodiments, the program code involved in the embodiments of the present application can be deployed and executed on one server, or on multiple servers located in one location, or on multiple servers distributed in multiple locations and interconnected through a communication network. Multiple servers distributed in multiple locations and interconnected through a communication network can constitute a blockchain system.
[0218] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.
[0219] The above are only optional embodiments of the embodiments of the present application and are not intended to limit the embodiments of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of this application.
Claims
1. A method for generating a face image, characterized in that: The method comprises: Acquire a facial image of a first face, acquire a facial image of a second face, and acquire facial vertex data, wherein the facial vertex data includes facial vertex data of multiple faces with different facial shapes and different expressions; Using the first facial shape feature and the first expression feature as weighting coefficients, fitting the facial vertex data of the multiple faces, taking the first face as a fitting target, iteratively updating the first facial shape feature and the first expression feature so that the fitting result approaches the first face, and determining the first facial shape feature and the first expression feature obtained from the last update as the target facial shape feature and the target expression feature of the first face, respectively; Using the second facial shape feature and the second facial expression feature as weighting coefficients, fitting the facial vertex data of the multiple faces, taking the second face as a fitting target, iteratively updating the second facial shape feature and the second facial expression feature so that the fitting result approaches the second face, and determining the second facial shape feature and the second facial expression feature obtained from the last update as the target facial shape feature and the target facial expression feature of the second face, respectively; A target facial image of the second face is generated based on the target expression feature of the first face, the target facial shape feature of the second face, and the target facial texture feature of the second face.
2. The method according to claim 1, characterized in that The method of fitting the facial vertex data of the multiple faces using the first facial shape feature and the first facial expression feature as weighting coefficients, taking the first face as a fitting target, and iteratively updating the first facial shape feature and the first facial expression feature so that the fitting result approaches the first face includes: Based on the facial image of the first face, acquiring facial vertex data of the first face; Using a preset first facial shape feature and a preset first expression feature as weighting coefficients, fitting facial vertex data of the multiple faces; determining a fitting error based on the fitting result and the facial vertex data of the first face; The preset first facial shape feature and the preset first expression feature are iteratively updated to reduce the fitting error.
3. The method according to claim 2, characterized in that The facial vertex data of the first face is two-dimensional data, and the facial vertex data of the multiple faces is three-dimensional data; determining the fitting error based on the fitting result and the facial vertex data of the first face includes: Projecting the three-dimensional facial vertex data in the fitting result into a two-dimensional coordinate system to obtain two-dimensional facial vertex data; The difference between the obtained two-dimensional facial vertex data and the facial vertex data of the first face is determined as the fitting error.
4. The method according to claim 2, characterized in that The facial vertex data of the first face is facial key point data of the first face; The method of fitting the facial vertex data of the plurality of faces by using the preset first facial shape feature and the preset first facial expression feature as weighting coefficients includes: Determining, based on the facial key point data of the first face, facial key point data of the multiple faces that match the facial key point data of the first face; The preset first face shape feature and the preset first expression feature are used as weighting coefficients to fit the facial key point data of the multiple faces.
5. The method according to claim 1, wherein The method of fitting the facial vertex data of the multiple faces using the first facial shape feature and the first facial expression feature as weighting coefficients, taking the first face as a fitting target, iteratively updating the first facial shape feature and the first facial expression feature so that the fitting result approaches the first face, and determining the first facial shape feature and the first facial expression feature obtained from the last update as the target facial shape feature and the target facial expression feature of the first face, respectively, includes: Using the first facial shape feature, the first facial expression feature, and the first facial rigid transformation matrix as weighting coefficients, fitting facial vertex data of the multiple faces, taking the first face as a fitting target, iteratively updating the first facial shape feature, the first facial expression feature, and the first facial rigid transformation matrix so that the fitting result approaches the first face, and determining the first facial shape feature and the first facial expression feature obtained from the last update as the target facial shape feature and the target facial expression feature of the first face, respectively. The facial rigid transformation matrix is used to translate and rotate the face; Decomposing the target pose angle feature of the first face from the face rigid transformation matrix obtained by the last update; Generating a target facial image of the second face based on the target expression feature of the first face, the target facial shape feature of the second face, and the target facial texture feature of the second face includes: A target facial image of the second face is generated based on the target expression feature of the first face, the target posture angle feature of the first face, the target facial shape feature of the second face, and the target facial texture feature of the second face.
6. The method according to claim 5, characterized in that The method of fitting the facial vertex data of the multiple faces using the first facial shape feature, the first facial expression feature, and the first facial rigid transformation matrix as weighting coefficients, taking the first face as a fitting target, and iteratively updating the first facial shape feature, the first facial expression feature, and the first facial rigid transformation matrix so that the fitting result approaches the first face includes: Fitting the facial vertex data of the multiple faces based on a preset first facial shape feature and a preset first expression feature to obtain a first fitting result; Determining a first facial rigid transformation matrix based on the first fitting result and facial vertex data of the first face; updating the preset first facial shape feature and the preset first expression feature based on the first facial rigid transformation matrix, the first fitting result, and the facial vertex data of the first face to obtain an updated first facial shape feature and an updated first expression feature; Fitting the facial vertex data of the multiple faces based on the updated first facial shape feature and the updated first facial expression feature to obtain a second fitting result; updating the first face rigid transformation matrix based on the second fitting result and the facial vertex data of the first face; The first face rigid transformation matrix, the first face shape feature, and the first expression feature are updated alternately until an iteration termination condition is reached.
7. The method according to claim 1, characterized in that The acquiring of face vertex data includes: Obtaining a first number of groups of facial vertex data, where one group of facial vertex data corresponds to one face, and the first number of groups of facial vertex data is facial vertex data obtained when a third number of faces make a second number of expressions; Perform dimensionality reduction on the first number of groups of facial vertex data to obtain a fourth number of groups of facial vertex data, where the fourth number is smaller than the first number.
8. The method according to claim 7, characterized in that The performing dimensionality reduction on the first number of groups of face vertex data to obtain a fourth number of groups of face vertex data includes: Multiple groups of facial vertex data that meet similar conditions in the first number of groups of facial vertex data are fused into one group of facial vertex data to obtain the fourth number of groups of facial vertex data.
9. The method according to claim 1, characterized in that The facial image of the first face is a facial image of the first face in any video frame in the reference video, and the second face is a face of a virtual object; the method further includes: For the facial images of the first face in the multiple video frames of the reference video, sequentially performing the steps of generating the target facial image of the second face, to obtain facial images of the virtual face corresponding to the multiple video frames respectively; Based on the order of the multiple video frames in the reference video, facial images of the virtual faces corresponding to the multiple video frames are displayed in sequence at the face position of the virtual object.
10. A facial image generation device, characterized in that: The device comprises: an acquisition module, configured to acquire a facial image of a first face, acquire a facial image of a second face, and acquire facial vertex data, wherein the facial vertex data includes facial vertex data of multiple faces with different facial shapes and different expressions; a fitting module, configured to fit the facial vertex data of the multiple faces using the first facial shape feature and the first facial expression feature as weighting coefficients, taking the first face as a fitting target, iteratively updating the first facial shape feature and the first facial expression feature so that a fitting result approaches the first face, and determining the first facial shape feature and the first facial expression feature obtained from the last update as the target facial shape feature and the target facial expression feature of the first face, respectively; The fitting module is further configured to use the second facial shape feature and the second facial expression feature as weighting coefficients to fit the facial vertex data of the multiple faces, take the second face as a fitting target, iteratively update the second facial shape feature and the second facial expression feature so that the fitting result approaches the second face, and determine the second facial shape feature and the second facial expression feature obtained from the last update as the target facial shape feature and the target facial expression feature of the second face, respectively; A generating module is used to generate a target facial image of the second face based on the target expression feature of the first face, the target facial shape feature of the second face and the target facial texture feature of the second face.
11. A terminal, characterized in that: The terminal includes a processor and a memory, wherein the memory stores at least one program code, and the at least one program code is loaded and executed by the processor to implement the operations performed in the facial image generation method according to any one of claims 1 to 9.
12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one program code, and the at least one program code is loaded and executed by the processor to implement the operations performed in the facial image generation method according to any one of claims 1 to 9.
