Three-dimensional face reconstruction method, device, electronic device and storage medium

Abstract

The present application discloses a method, device, electronic device and storage medium for reconstructing a three-dimensional face, and relates to the technical field of electronic devices. The method comprises: obtaining shape parameters and texture parameters of a face to be reconstructed, inputting the shape parameters into a first model, obtaining a three-dimensional face shape output by the first model, inputting the texture parameters into a second model, obtaining a face texture map output by the second model, wherein at least one of the first model and the second model is obtained based on training of a generative adversarial network, and a target three-dimensional face is generated based on the three-dimensional face shape and the face texture map, wherein the target three-dimensional face includes texture information generated based on the face texture map. The present application generates a three-dimensional face shape and / or a face texture map through a trained generative adversarial network, so that the generated target three-dimensional face has rich detail features, thereby improving the reconstruction effect of the three-dimensional face.

Description

Technical Field

[0001] The present application relates to the technical field of electronic devices, and more specifically, to a three-dimensional face reconstruction method, device, electronic device and storage medium. Background Art

[0002] In the field of computer vision and computer graphics, 3D face reconstruction is a hot topic that is gradually gaining popularity, as it can be widely used in face recognition, face editing, human-computer interaction, expression-driven animation, augmented reality, virtual reality and other fields. 3D face reconstruction based on a single RGB image, that is, using a single RGB image to reconstruct the 3D face shape and texture, is an enduring topic. At present, the details of the 3D face reconstructed based on a single RGB image are poor and lack realism. Summary of the invention

[0003] In view of the above problems, the present application proposes a three-dimensional face reconstruction method, device, electronic device and storage medium to solve the above problems.

[0004] In a first aspect, an embodiment of the present application provides a method for reconstructing a three-dimensional face, the method comprising: obtaining shape parameters and texture parameters of a face to be reconstructed; inputting the shape parameters into a first model to obtain a three-dimensional face shape output by the first model; inputting the texture parameters into a second model to obtain a face texture map output by the second model, wherein at least one of the first model and the second model is obtained based on generative adversarial network training; generating a target three-dimensional face based on the three-dimensional face shape and the face texture map, wherein the target three-dimensional face includes texture information generated based on the face texture map.

[0005] In a second aspect, an embodiment of the present application provides a three-dimensional face reconstruction device, the device comprising: a parameter acquisition module, used to acquire shape parameters and texture parameters of a face to be reconstructed; a face shape acquisition module, used to input the shape parameters into a first model to obtain a three-dimensional face shape output by the first model; a texture map acquisition module, used to input the texture parameters into a second model to obtain a face texture map output by the second model, wherein at least one of the first model and the second model is obtained based on generative adversarial network training; a three-dimensional face generation module, used to generate a target three-dimensional face based on the three-dimensional face shape and the face texture map, wherein the target three-dimensional face includes texture information generated based on the face texture map.

[0006] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor, the memory stores instructions, and when the instructions are executed by the processor, the processor executes the above method.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored, and the program code can be called by a processor to execute the above method.

[0008] The three-dimensional face reconstruction method, device, electronic device and storage medium provided in the embodiments of the present application obtain shape parameters and texture parameters of the face to be reconstructed, input the shape parameters into a first model to obtain the three-dimensional face shape output by the first model, input the texture parameters into a second model to obtain the face texture map output by the second model, wherein at least one of the first model and the second model is obtained based on generative adversarial network training, and a target three-dimensional face is generated based on the three-dimensional face shape and the face texture map, wherein the target three-dimensional face includes texture information generated based on the face texture map, so that the three-dimensional face shape and / or the face texture map are generated by the trained generative adversarial network, so that the generated target three-dimensional face has rich detail features, thereby improving the reconstruction effect of the three-dimensional face. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0010] Figure 1 A schematic diagram showing a flow chart of a method for reconstructing a three-dimensional face provided by an embodiment of the present application;

[0011] Figure 2 A schematic diagram showing a flow chart of a method for reconstructing a three-dimensional face provided by another embodiment of the present application;

[0012] Figure 3 A schematic diagram showing a flow chart of a method for reconstructing a three-dimensional face provided in yet another embodiment of the present application;

[0013] Figure 4 This application shows Figure 3 The flowchart of step S306 of the method for reconstructing a three-dimensional face is shown;

[0014] Figure 5 A schematic diagram showing a flow chart of a method for reconstructing a three-dimensional face provided in another embodiment of the present application;

[0015] Figure 6 A schematic flow chart of a method for reconstructing a three-dimensional face provided in yet another embodiment of the present application is shown;

[0016] Figure 7 A schematic diagram showing a flow chart of a method for reconstructing a three-dimensional face provided in yet another embodiment of the present application;

[0017] Figure 8 A schematic flow chart of a method for reconstructing a three-dimensional face provided in yet another embodiment of the present application is shown;

[0018] Fig. 9 A module block diagram of a three-dimensional face reconstruction device provided in an embodiment of the present application is shown;

[0019] Fig.10 A block diagram of an electronic device for executing a method for reconstructing a three-dimensional face according to an embodiment of the present application is shown;

[0020] Fig.11 A storage unit for storing or carrying program codes for implementing a three-dimensional face reconstruction method according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0022] At present, the classic method for 3D face reconstruction based on a single RGB image is the 3D morphable model (3DMM), whose basic idea is to collect 3D faces in advance and complete the establishment of a database. The shape vector, expression vector and texture vector of the face are subjected to principal component analysis, and these vectors are converted into independent representations. On this basis, any face can be represented by a linear combination of the average face and the principal components. Through research, the inventors found that there are two important but often overlooked problems in 3DMM: 1. After collecting the original three-dimensional face scanning data, the common practice is to use a pre-made template face to align the original face data. However, due to the low quality of the template face (small number of vertices and faces), the aligned face will "smooth" the original face data, thereby losing some face detail information (for example, acne, wrinkles, etc.); 2. Existing 3DMM-based methods all start with an average face, and the three-dimensional face shape features are obtained by the average face plus the linear combination of the principal components. The three-dimensional face shape features obtained by this method will be more regularized, so that the generated face tends to be an average face, and the realism is poor.

[0023] In view of the above problems, the inventors have discovered after long-term research and proposed the three-dimensional face reconstruction method, device, electronic device and storage medium provided in the embodiments of the present application, which generates a three-dimensional face shape and / or face texture map through a trained generative adversarial network, so that the generated target three-dimensional face has rich detail features, thereby improving the reconstruction effect of the three-dimensional face. The specific three-dimensional face reconstruction method is described in detail in the subsequent embodiments.

[0024] See also Figure 1 , Figure 1 The flowchart of a 3D face reconstruction method provided by an embodiment of the present application is shown. The 3D face reconstruction method is used to generate a 3D face shape and / or face texture map through a trained generative adversarial network, so that the generated target 3D face has rich detail features, thereby improving the reconstruction effect of the 3D face. In a specific embodiment, the 3D face reconstruction method is applied to Fig. 9 The three-dimensional face reconstruction device 200 and the electronic device 100 equipped with the three-dimensional face reconstruction device 200 are shown in FIG. Fig.10 ). The following will take an electronic device as an example to illustrate the specific process of this embodiment. Of course, it can be understood that the electronic device used in this embodiment can be a smart phone, a tablet computer, a desktop computer, a wearable electronic device, etc., which is not limited here. Figure 1 The process shown in the figure is described in detail, and the three-dimensional face reconstruction method may specifically include the following steps:

[0025] Step S101: Obtain shape parameters and texture parameters of the face to be reconstructed.

[0026] In this embodiment, the face to be reconstructed may be a three-dimensional face that needs to be reconstructed. Then obtaining the shape parameters and texture parameters of the face to be reconstructed may include: obtaining the shape parameters and texture parameters of the three-dimensional face that needs to be reconstructed. Among them, the shape parameters may include appearance parameters and expression parameters, and obtaining the shape parameters and texture parameters of the face to be reconstructed may include: obtaining the appearance parameters, expression parameters and texture parameters of the three-dimensional face that needs to be reconstructed.

[0027] As a method, a two-dimensional color face can be captured by an RGB camera, and the captured two-dimensional color face can be processed to obtain a three-dimensional face corresponding to the two-dimensional color face as the face to be reconstructed, and the shape parameters and texture parameters of the face to be reconstructed can be obtained.

[0028] In some implementations, after obtaining the shape parameters of the face to be reconstructed, the shape parameters may be initialized, and after obtaining the texture parameters of the face to be reconstructed, the texture parameters may be initialized.

[0029] Step S102: inputting the shape parameters into a first model to obtain a three-dimensional face shape output by the first model.

[0030] In some embodiments, the first model and the second model may be pre-trained and set, wherein at least one of the first model and the second model is obtained based on training of a generative adversarial network. Therefore, as one approach, the first model may be obtained based on training of a generative adversarial network, and the second model may be obtained based on training of a generative adversarial network. As another approach, the first model may be obtained based on training of a generative adversarial network, and the second model may be a principal component analysis statistical model. As another approach, the first model may be a principal component analysis statistical model, and the second model may be obtained based on training of a generative adversarial network.

[0031] In this embodiment, after obtaining the shape parameters of the face to be reconstructed, the shape parameters can be input into the first model to obtain the three-dimensional face shape output by the first model. When the first model is a trained generative adversarial network, the shape parameters can be input into the trained generative adversarial network to obtain the three-dimensional face shape output by the trained generative adversarial network, and when the first model is a principal component analysis statistical model, the shape parameters can be input into the principal component analysis statistical model to obtain the three-dimensional face shape output by the principal component analysis statistical model.

[0032] Step S103: input the texture parameters into a second model to obtain a face texture map output by the second model, wherein at least one of the first model and the second model is obtained based on generative adversarial network training.

[0033] In this embodiment, after obtaining the texture parameters of the face to be reconstructed, the texture parameters can be input into the second model to obtain the face texture map output by the second model. When the second model is a trained generative adversarial network, the texture parameters can be input into the trained generative adversarial network to obtain the face texture map output by the trained generative adversarial network; when the second model is a principal component analysis statistical model, the texture parameters can be input into the principal component analysis statistical model to obtain the face texture map output by the principal component analysis statistical model.

[0034] Step S104: generating a target three-dimensional face based on the three-dimensional face shape and the face texture map, wherein the target three-dimensional face includes texture information generated based on the face texture map.

[0035] In this embodiment, after obtaining the three-dimensional face shape and the face texture map, a target three-dimensional face can be generated based on the three-dimensional face shape and the face texture map, wherein the target three-dimensional face includes texture information generated based on the face texture map, that is, the generated target three-dimensional face is a three-dimensional face with texture.

[0036] In some implementations, after obtaining the three-dimensional face shape and the face texture map, the three-dimensional face shape and the face texture map may be combined to generate a target three-dimensional face.

[0037] An embodiment of the present application provides a method for reconstructing a three-dimensional face, which obtains shape parameters and texture parameters of a face to be reconstructed, inputs the shape parameters into a first model, obtains a three-dimensional face shape output by the first model, and inputs the texture parameters into a second model, obtains a face texture map output by the second model, wherein at least one of the first model and the second model is obtained based on generative adversarial network training, and a target three-dimensional face is generated based on the three-dimensional face shape and the face texture map, wherein the target three-dimensional face includes texture information generated based on the face texture map, so that the three-dimensional face shape and / or the face texture map are generated by the trained generative adversarial network, so that the generated target three-dimensional face has rich detail features, thereby improving the reconstruction effect of the three-dimensional face.

[0038] See also Figure 2 , Figure 2 A schematic diagram of a process flow of a method for reconstructing a three-dimensional face provided by another embodiment of the present application is shown. Figure 2 The process shown is described in detail. In this embodiment, the first model is a trained generative adversarial network, and the three-dimensional face reconstruction method can specifically include the following steps:

[0039] Step S201: Obtain shape parameters and texture parameters of the face to be reconstructed.

[0040] For the detailed description of step S201, please refer to step S101, which will not be repeated here.

[0041] Step S202: inputting the shape parameters into the trained generative adversarial network to obtain a face coordinate map output by the trained generative adversarial network.

[0042] In this embodiment, the first model is a trained generative adversarial network. As a method, a training data set may be first collected, wherein the attributes or features of one type of data in the training data set are different from those of another type of data, and then the generative adversarial network is trained and modeled by using the collected training data set according to a preset algorithm, thereby summarizing the rules based on the training data set to obtain a trained generative adversarial network. In this embodiment, the training data set may be, for example, a plurality of shape parameters and a plurality of face coordinate maps having a corresponding relationship.

[0043] In some implementations, after obtaining shape parameters of the face to be reconstructed, the shape parameters may be input into a trained generative adversarial network to obtain a face coordinate map output by the trained generative adversarial network.

[0044] Step S203: obtaining the three-dimensional face shape based on the face coordinate map.

[0045] In this embodiment, after obtaining the face coordinate map, the three-dimensional face shape can be obtained based on the face coordinate map. In some embodiments, after obtaining the face coordinate map, the coordinates of each vertex of the three-dimensional face can be obtained based on the face coordinate map, and the three-dimensional face shape can be obtained based on the coordinates of each vertex of the three-dimensional face.

[0046] Step S204: input the texture parameters into a second model to obtain a face texture map output by the second model.

[0047] Step S205: generating a target three-dimensional face based on the three-dimensional face shape and the face texture map, wherein the target three-dimensional face includes texture information generated based on the face texture map.

[0048] For the detailed description of step S204 to step S205, please refer to step S103 to step S104, which will not be repeated here.

[0049] Another embodiment of the present application provides a method for reconstructing a three-dimensional face. Figure 1 The three-dimensional face reconstruction method shown also obtains a face coordinate map based on shape parameters through a trained generative adversarial network, and obtains a three-dimensional face shape based on the face coordinate map, thereby improving the accuracy of the obtained three-dimensional face shape.

[0050] See also Figure 3 , Figure 3 A schematic diagram of a process flow of a method for reconstructing a three-dimensional face provided by another embodiment of the present application is shown below. Figure 3 The process shown in the figure is described in detail, and the three-dimensional face reconstruction method may specifically include the following steps:

[0051] Step S301: Obtain shape parameters and texture parameters of the face to be reconstructed.

[0052] Step S302: input the shape parameters into a first model to obtain a three-dimensional face shape output by the first model.

[0053] Step S303: input the texture parameters into a second model to obtain a face texture map output by the second model, wherein at least one of the first model and the second model is obtained based on generative adversarial network training.

[0054] Step S304: generating a target three-dimensional face based on the three-dimensional face shape and the face texture map, wherein the target three-dimensional face includes texture information generated based on the face texture map.

[0055] For the detailed description of steps S301 to S304 , please refer to steps S101 to S104 , which will not be repeated here.

[0056] Step S305: Rendering the target three-dimensional face based on the rendering function, camera parameters and illumination parameters to generate a two-dimensional rendered face.

[0057] In this embodiment, after obtaining the target three-dimensional human face, the target three-dimensional human face may be rendered based on the rendering function, camera parameters, and illumination parameters to generate a two-dimensional rendered human face.

[0058] In some implementations, the camera position [x c ,y c ,z c ], Camera orientation and focal length f c Parameterized camera parameters p c =[x c ,y c ,z c ,x' c ,y' c ,z' c ]; Based on the illumination model, the point light source position [x l ,y l ,z l ]、Color[r l ,g l ,b l ] and ambient light color [r a ,g a ,b a ] Parameterized lighting parameters p l =[x l ,y l ,z l ,r l ,g l ,b l ,r a ,g a ,b a Finally, the entire image rendering process can be done by to complete, where S and g represent the first model and the second model respectively. For example, S represents the first generative adversarial network, g represents the second generative adversarial network, p represents a function that maps the three-dimensional face shape to spatial coordinates and maps the face texture map to texture coordinates, Represents a rendering function that renders a 3D face into a 2D image, which can be a weak perspective projection or a perspective projection. Therefore, as a way, it can be based on the rendering function Camera parameter p c And the lighting parameter p l The target three-dimensional face (based on the three-dimensional face shape and face texture map) is rendered to generate a two-dimensional rendered face.

[0059] Step S306: Acquire the posture of the face to be reconstructed.

[0060] In this embodiment, the posture of the face to be reconstructed can be obtained. As an implementable method, key point detection can be performed on the face to be reconstructed to obtain the facial key points corresponding to the face to be reconstructed, and the posture of the face to be reconstructed can be obtained based on the facial key points.

[0061] See also Figure 4 , Figure 4 This application shows Figure 3 The flowchart of step S306 of the method for reconstructing a three-dimensional face is shown in FIG. Figure 4 The process shown is described in detail, and the method may specifically include the following steps:

[0062] Step S3061: performing key point detection on the face to be reconstructed, and obtaining a plurality of three-dimensional face key points of the face to be reconstructed as first face key points.

[0063] In some implementations, key point detection may be performed on the face to be reconstructed, and multiple three-dimensional face key points of the face to be reconstructed are obtained as first face key points. In some implementations, key point detection may be performed on the face to be reconstructed, and multiple three-dimensional face key points of the face to be reconstructed are obtained as first face key points, and multiple two-dimensional face key points of the face to be reconstructed are obtained as second face key points.

[0064] In this embodiment, a face to be reconstructed can be obtained, and an advanced, pre-trained face key point detector is used to detect and obtain multiple two-dimensional face key points as the second face key points and multiple three-dimensional face key points as the first face key points. As a method, a face to be reconstructed can be obtained, and an advanced, pre-trained face key point detector is used to detect and obtain 68 two-dimensional face key points. and 68 3D facial key points As a way to obtain the face to be reconstructed, and detect 68 two-dimensional face key points through an advanced, pre-trained face key point detector Then back-project 68 2D facial key points Generate 68 3D facial key points

[0065] Step S3062: Based on the first facial key points, the posture of the face to be reconstructed is obtained.

[0066] In some embodiments, after obtaining the first facial key point, the posture of the face to be reconstructed can be obtained based on the first facial key point. As a method, after obtaining the first facial key point, the Euler angle can be calculated based on the first facial key point, and the posture of the face to be reconstructed can be determined based on the Euler angle, wherein the Euler angle includes a roll angle, a pitch angle, and a yaw angle.

[0067] Step S307: Based on the relationship between the posture and the preset posture, the key point losses of the face to be reconstructed and the target three-dimensional face are calculated.

[0068] In some embodiments, a preset posture may be pre-set and stored, and the preset posture is used as a basis for judging the posture of the face to be reconstructed. Therefore, in this embodiment, after obtaining the posture of the face to be reconstructed, the posture may be compared with the preset posture to obtain the relationship between the posture and the preset posture, and based on the relationship between the posture and the preset posture, the key point loss of the face to be reconstructed and the target three-dimensional face is calculated.

[0069] In some embodiments, a preset roll angle, a preset pitch angle, and a preset yaw angle may be preset and stored, wherein the preset roll angle is used as a basis for determining the roll angle of the face to be reconstructed, the preset pitch angle is used as a basis for determining the pitch angle of the face to be reconstructed, and the preset yaw angle is used as a basis for determining the yaw angle of the face to be reconstructed. As a method, when the roll angle is greater than the preset roll angle, the pitch angle is greater than the preset pitch angle, and / or the yaw angle is greater than the preset yaw angle, it can be determined that the posture is greater than the preset posture, and when the roll angle is greater than the preset roll angle, the pitch angle is not greater than the preset pitch angle, and the yaw angle is not greater than the preset yaw angle, it can be determined that the posture is not greater than the preset posture.

[0070] Therefore, in this embodiment, after obtaining the first facial key point, the roll angle of the face to be reconstructed can be calculated based on the first facial key point, and the roll angle can be compared with the preset roll angle to obtain the relationship between the roll angle and the preset roll angle, and based on the relationship between the roll angle and the preset roll angle, the key point loss of the face to be reconstructed and the target three-dimensional face is calculated; after obtaining the first facial key point, the pitch angle of the face to be reconstructed can be calculated based on the first facial key point, and the pitch angle can be compared with the preset pitch angle to obtain the relationship between the pitch angle and the preset pitch angle, and based on the relationship between the pitch angle and the preset pitch angle, the key point loss of the face to be reconstructed and the target three-dimensional face is calculated; after obtaining the first facial key point, the yaw angle of the face to be reconstructed can be calculated based on the first facial key point, and the yaw angle can be compared with the preset yaw angle to obtain the relationship between the yaw angle and the preset yaw angle, and based on the relationship between the yaw angle and the preset yaw angle, the key point loss of the face to be reconstructed and the target three-dimensional face is calculated.

[0071] In some embodiments, when the posture representing the relationship between the posture of the face to be reconstructed and the preset posture is greater than the preset posture, multiple three-dimensional face key points of the target three-dimensional face can be obtained as the third face key points, and the key point loss of the first face key points and the third face key points can be calculated. As a way, when the posture representing the relationship between the posture of the face to be reconstructed and the preset posture is greater than the preset posture, key point detection can be performed on the target three-dimensional face, multiple three-dimensional face key points can be obtained as the third face key points, and the key point loss of the first face key points and the third face key points can be calculated.

[0072] In some embodiments, when the posture representing the relationship between the posture of the face to be reconstructed and the preset posture is not greater than the preset posture, multiple two-dimensional face key points corresponding to multiple three-dimensional face key points of the target three-dimensional face can be obtained as the fourth face key point, and the key point loss of the second face key point and the fourth face key point can be calculated. As a method, when the posture representing the relationship between the posture of the face to be reconstructed and the preset posture is not greater than the preset posture, key point detection can be performed on the target three-dimensional face to obtain multiple three-dimensional face key points, and the multiple three-dimensional face key points are projected to the pixel coordinate system according to the camera parameters to obtain the corresponding multiple two-dimensional face key points as the fourth face key point, and the key point loss of the second face key point and the fourth face key point is calculated.

[0073] In some embodiments, based on L lan =||M(I 0 )-M'(p s ,p e ,p c )||2 Calculate the key point loss of the face to be reconstructed and the target 3D face. In the above formula, M(I 0) is the first key point or the second key point corresponding to the image to be reconstructed, M'(p s ,p e ,p c ) is the coordinate of the projection of the third key point or the fourth key point corresponding to the target three-dimensional image, and || ||2 represents the L2 norm.

[0074] Step S308: iteratively optimize the shape parameters based on the key point loss to obtain first optimized shape parameters.

[0075] In this embodiment, after the key point loss is calculated, the shape parameters can be iteratively optimized based on the key point loss to obtain the first optimized shape parameters. As an implementation method, after the key point loss is calculated, the gradient of the key point loss relative to the shape parameter can be calculated using a gradient descent algorithm, thereby iteratively optimizing the shape parameters, wherein the iterative optimization process is continuously performed until the key point loss is almost converged, or until the number of iterations reaches a threshold number of times, thereby completing the iterative optimization of the shape parameters and obtaining the first optimized shape parameters.

[0076] Step S309: inputting the first optimized shape parameters into the first model as new shape parameters.

[0077] In some embodiments, after obtaining the first optimized shape parameters, the first optimized shape parameters can be input into the first model as new shape parameters to obtain a new three-dimensional face shape through the first model, and a new target three-dimensional face is obtained through the new three-dimensional face shape, thereby improving the accuracy of the obtained three-dimensional face.

[0078] Step S310: iteratively optimizing the camera parameters based on the key point loss to obtain optimized camera parameters.

[0079] In this embodiment, after the key point loss is calculated, the camera can be iteratively optimized based on the key point loss to obtain optimized camera parameters. As an implementation method, after the key point loss is calculated, the gradient of the key point loss relative to the camera parameters can be calculated using a gradient descent algorithm, thereby iteratively optimizing the camera parameters, wherein the iterative optimization process is continuously performed until the key point loss is almost converged, or until the number of iterations reaches a threshold, thereby completing the iterative optimization of the camera parameters and obtaining the optimized camera parameters.

[0080] Step S311: using the optimized camera parameters as new camera parameters to render the target three-dimensional human face.

[0081] In some implementations, after obtaining the optimized camera parameters, the optimized camera parameters may be used as new camera parameters to render the target three-dimensional face, obtain a new two-dimensional rendered face, and improve the accuracy of the obtained two-dimensional rendered face.

[0082] Another embodiment of the present application provides a method for reconstructing a three-dimensional face. Figure 1 The three-dimensional face reconstruction method shown in the figure also calculates the key point losses of the face to be reconstructed and the target three-dimensional face, and optimizes the shape parameters and camera parameters based on the key point losses to improve the realism of the target three-dimensional face.

[0083] See also Figure 5 , Figure 5 A schematic diagram of a process flow of a method for reconstructing a three-dimensional face provided by another embodiment of the present application is shown. Figure 5 The process shown in the figure is described in detail, and the three-dimensional face reconstruction method may specifically include the following steps:

[0084] Step S401: Obtain shape parameters and texture parameters of the face to be reconstructed.

[0085] Step S402: input the shape parameters into a first model to obtain a three-dimensional face shape output by the first model.

[0086] Step S403: input the texture parameters into a second model to obtain a face texture map output by the second model, wherein at least one of the first model and the second model is obtained based on generative adversarial network training.

[0087] Step S404: generating a target three-dimensional face based on the three-dimensional face shape and the face texture map, wherein the target three-dimensional face includes texture information generated based on the face texture map.

[0088] For the detailed description of steps S401 to S404, please refer to steps S101 to S104, which will not be repeated here.

[0089] Step S405: Rendering the target three-dimensional face based on the rendering function, camera parameters and illumination parameters to generate a two-dimensional rendered face.

[0090] For the detailed description of step S405, please refer to step S305, which will not be repeated here.

[0091] Step S406: obtaining biometric information of the face to be reconstructed as first biometric information, and obtaining biometric information of the two-dimensionally rendered face as second biometric information.

[0092] In some embodiments, an advanced face recognition network can be used to capture the biometric information of the face to be reconstructed, and the biometric information of the face to be reconstructed can be obtained as the first biometric information, and an advanced face recognition network can be used to capture the biometric information of the two-dimensional rendered face, and the biometric information of the two-dimensional rendered face can be obtained as the second biometric information. The biometric information can include the fatness of the face, the length of the eyebrows, the size of the eyes, the height of the nose, the thickness of the lips, etc., which are not limited here. The face recognition network can be F n (I): Among them, n means there are n convolutional layers.

[0093] Step S407: Calculate the biometric loss of the face to be reconstructed and the two-dimensional rendered face based on the first biometric information and the second biometric information.

[0094] In this embodiment, after obtaining the first biometric information of the face to be reconstructed and the second biometric information of the two-dimensionally rendered face, the biometric loss of the face to be reconstructed and the two-dimensionally rendered face can be calculated based on the first biometric information and the second biometric information. In some embodiments, after obtaining the first biometric information of the face to be reconstructed and the second biometric information of the two-dimensionally rendered face, the first face feature vector of the first biometric information and the second face feature vector of the second biometric information can be calculated, and then the biometric loss of the face to be reconstructed and the two-dimensionally rendered face can be calculated based on the first face feature vector of the first biometric information and the second face feature vector of the second biometric information.

[0095] In some embodiments, the Calculate the biometric loss of the face to be reconstructed and the two-dimensional rendered face. In the above formula, F n (I 0 ) represents the first facial feature vector of the face to be reconstructed, F n (I R ) represents the second facial feature vector of the two-dimensional rendered face.

[0096] Step S408: iteratively optimizing the shape parameters and the texture parameters based on the biometric loss to obtain second optimized shape parameters and first optimized texture parameters.

[0097] In this embodiment, after the biometric loss is calculated, the shape parameters can be iteratively optimized based on the biometric loss to obtain the second optimized shape parameters. As an implementation method, after the biometric loss is calculated, the gradient of the biometric loss relative to the shape parameters can be calculated using a gradient descent algorithm, thereby iteratively optimizing the shape parameters, wherein the iterative optimization process is continuously performed until the key point loss is almost converged, or until the number of iterations reaches a threshold, thereby completing the iterative optimization of the shape parameters and obtaining the second optimized shape parameters.

[0098] In this embodiment, after the biometric loss is calculated, the texture parameters can be iteratively optimized based on the biometric loss to obtain the first optimized texture parameters. As an implementation method, after the biometric loss is calculated, the gradient of the biometric loss relative to the texture parameters can be calculated using a gradient descent algorithm, thereby iteratively optimizing the texture parameters, wherein the iterative optimization process is continuously performed until the key point loss is almost converged, or until the number of iterations reaches a threshold, thereby completing the iterative optimization of the texture parameters and obtaining the first optimized texture parameters.

[0099] Step S409: inputting the second optimized shape parameters as new shape parameters into the first model, and inputting the first optimized texture parameters as new texture parameters into the second model.

[0100] In some embodiments, after obtaining the second optimized shape parameters, the second optimized shape parameters can be input into the first model as new shape parameters to obtain a new three-dimensional face shape through the first model, and a new target three-dimensional face is obtained through the new three-dimensional face shape, thereby improving the accuracy of the obtained three-dimensional face.

[0101] In some embodiments, after obtaining the first optimized texture parameters, the first optimized texture parameters can be input into the second model as new texture parameters to obtain new facial texture coordinates through the second model, and a new target three-dimensional face can be obtained through the new facial texture coordinates to improve the accuracy of the obtained three-dimensional face.

[0102] Another embodiment of the present application provides a method for reconstructing a three-dimensional face. Figure 1 The three-dimensional face reconstruction method shown in the figure also improves the realism of the target three-dimensional face by calculating the biometric loss of the face to be reconstructed and the two-dimensional rendered face, and optimizing the shape parameters and camera parameters based on the biometric loss.

[0103] See also Figure 6 , Figure 6 FIG. 1 is a flow chart showing a method for reconstructing a three-dimensional face provided by yet another embodiment of the present application. Figure 6The process shown in the figure is described in detail, and the three-dimensional face reconstruction method may specifically include the following steps:

[0104] Step S501: Obtain shape parameters and texture parameters of the face to be reconstructed.

[0105] Step S502: input the shape parameters into a first model to obtain a three-dimensional face shape output by the first model.

[0106] Step S503: input the texture parameters into a second model to obtain a face texture map output by the second model, wherein at least one of the first model and the second model is obtained based on generative adversarial network training.

[0107] Step S504: generating a target three-dimensional face based on the three-dimensional face shape and the face texture map, wherein the target three-dimensional face includes texture information generated based on the face texture map.

[0108] For the detailed description of steps S501 to S504 , please refer to steps S101 to S104 , which will not be repeated here.

[0109] Step S505: Rendering the target three-dimensional face based on the rendering function, camera parameters and illumination parameters to generate a two-dimensional rendered face.

[0110] For the detailed description of step S505, please refer to step S305, which will not be repeated here.

[0111] Step S506: acquiring attribute content information of the to-be-reconstructed face as first attribute content information, and acquiring attribute content information of the two-dimensionally rendered face as second attribute content information.

[0112] In some implementations, the convolutional layer features in an advanced face recognition network can be used to capture the attribute content information of the face to be reconstructed, and the attribute content information of the face to be reconstructed can be obtained as the first attribute content information, and the convolutional layer in an advanced face recognition network can be used to capture the attribute content information of the two-dimensional rendered face, and the attribute content information of the two-dimensional rendered face can be obtained as the second attribute content information. The attribute content information may include expression, posture, lighting, etc., which are not limited here.

[0113] Step S507: Calculate the attribute content loss of the to-be-reconstructed face and the two-dimensionally rendered face based on the first attribute content information and the second attribute content information.

[0114] In this embodiment, after obtaining the first attribute content information of the face to be reconstructed and the second attribute content information of the two-dimensional rendered face, the attribute content loss of the face to be reconstructed and the two-dimensional rendered face can be calculated based on the first attribute content information and the second attribute content information.

[0115] In some embodiments, the Calculate the attribute content loss of the face to be reconstructed and the 2D rendered face. In the above formula, F j (I 0 ) represents the middle-level features of the face to be reconstructed, F j (I R ) represents the mid-level facial features of a 2D rendered face.

[0116] Step S508: iteratively optimizing the shape parameters and the texture parameters based on the attribute content loss to obtain third optimized shape parameters and second optimized texture parameters.

[0117] In this embodiment, after the attribute content loss is calculated, the shape parameters can be iteratively optimized based on the attribute content loss to obtain the third optimized shape parameters. As an implementation method, after the attribute content loss is calculated, the gradient of the attribute content loss relative to the shape parameters can be calculated using a gradient descent algorithm, thereby iteratively optimizing the shape parameters, wherein the iterative optimization process is continuously performed until the key point loss is almost converged, or until the number of iterations reaches a threshold number of iterations, thereby completing the iterative optimization of the shape parameters and obtaining the third optimized shape parameters.

[0118] In this embodiment, after calculating the attribute content loss, the texture parameters can be iteratively optimized based on the attribute content loss to obtain the second optimized texture parameters. As an implementation method, after calculating the attribute content loss, the gradient of the attribute content loss relative to the texture parameters can be calculated using a gradient descent algorithm, thereby iteratively optimizing the texture parameters, wherein the iterative optimization process is continuously performed until the key point loss is almost converged, or until the number of iterations reaches a threshold number of iterations, thereby completing the iterative optimization of the texture parameters and obtaining the second optimized texture parameters.

[0119] Step S509: input the third optimized shape parameters as new shape parameters into the first model, and input the second optimized texture parameters as new texture parameters into the second model.

[0120] In some embodiments, after obtaining the third optimized shape parameters, the third optimized shape parameters can be input into the first model as new shape parameters to obtain a new three-dimensional face shape through the first model, and a new target three-dimensional face is obtained through the new three-dimensional face shape, thereby improving the accuracy of the obtained three-dimensional face.

[0121] In some embodiments, after obtaining the second optimized texture parameters, the second optimized texture parameters can be input into the second model as new texture parameters to obtain new facial texture coordinates through the second model, and a new target three-dimensional face can be obtained through the new facial texture coordinates, thereby improving the accuracy of the obtained three-dimensional face.

[0122] Another embodiment of the present application provides a method for reconstructing a three-dimensional face. Figure 1 The three-dimensional face reconstruction method shown in the figure also calculates the attribute content information loss of the face to be reconstructed and the two-dimensional rendered face, and optimizes the shape parameters and camera parameters based on the attribute content information loss to improve the realism of the target three-dimensional face.

[0123] See also Figure 7 , Figure 7 A schematic diagram of a process flow of a method for reconstructing a three-dimensional face provided by another embodiment of the present application is shown. Figure 7 The process shown in the figure is described in detail, and the three-dimensional face reconstruction method may specifically include the following steps:

[0124] Step S601: Obtain shape parameters and texture parameters of the face to be reconstructed.

[0125] Step S602: input the shape parameters into a first model to obtain a three-dimensional face shape output by the first model.

[0126] Step S603: input the texture parameters into a second model to obtain a face texture map output by the second model, wherein at least one of the first model and the second model is obtained based on generative adversarial network training.

[0127] Step S604: generating a target three-dimensional face based on the three-dimensional face shape and the face texture map, wherein the target three-dimensional face includes texture information generated based on the face texture map.

[0128] For the detailed description of steps S601 to S604 , please refer to steps S101 to S104 , which will not be repeated here.

[0129] Step S605: Rendering the target three-dimensional face based on the rendering function, camera parameters and illumination parameters to generate a two-dimensional rendered face.

[0130] For the detailed description of step S605, please refer to step S305, which will not be repeated here.

[0131] Step S606: acquiring pixel information of the to-be-reconstructed face as first pixel information, and acquiring pixel information based on the two-dimensionally rendered face as second pixel information.

[0132] In some implementations, pixel information of the face to be reconstructed is obtained as the first pixel information, and pixel information of the two-dimensionally rendered face is obtained as the second pixel information. The pixel information of the face to be reconstructed and the pixel information of the two-dimensionally rendered face can be obtained by a pixel extraction module, which is not limited here.

[0133] Step S607: Calculate the pixel loss of the to-be-reconstructed face and the two-dimensionally rendered face based on the first pixel information and the second pixel information.

[0134] In this embodiment, after obtaining first pixel information of the face to be reconstructed and second pixel information of the two-dimensional rendered face, the pixel loss of the face to be reconstructed and the two-dimensional rendered face can be calculated based on the first pixel information and the second pixel information.

[0135] In some embodiments, based on L pix =||I 0 -I R ||1 Calculate the pixel loss of the face to be reconstructed and the two-dimensional rendered face. In the above formula, I 0 represents the face to be reconstructed, I R represents a two-dimensional rendered face, and || ||1 represents the L1 norm.

[0136] Step S608: iteratively optimizing the illumination parameters based on the pixel loss to obtain optimized illumination parameters.

[0137] In this embodiment, after the pixel loss is calculated, the illumination parameters can be iteratively optimized based on the pixel loss to obtain the optimized illumination parameters. As an implementation method, after the pixel loss is calculated, the gradient of the pixel loss relative to the illumination parameters can be calculated using a gradient descent algorithm, thereby iteratively optimizing the illumination parameters, wherein the iterative optimization process is continuously performed until the key point loss is almost converged, or until the number of iterations reaches a threshold, thereby completing the iterative optimization of the illumination parameters and obtaining the optimized illumination parameters.

[0138] In some implementations, after the pixel loss is calculated, the shape parameters and texture parameters may be iteratively optimized based on the pixel loss, which will not be described in detail here.

[0139] Step S609: using the optimized illumination parameters as new illumination parameters to render the target three-dimensional human face.

[0140] In some implementations, after obtaining the optimized illumination parameters, the optimized illumination parameters may be used as new illumination parameters to render the target three-dimensional face, obtain a new two-dimensional rendered face, and improve the accuracy of the obtained two-dimensional rendered face.

[0141] Another embodiment of the present application provides a method for reconstructing a three-dimensional face. Figure 1 The three-dimensional face reconstruction method shown in the figure also improves the realism of the target three-dimensional face by calculating the pixel loss of the face to be reconstructed and the two-dimensional rendered face, and optimizing the shape parameters and camera parameters based on the pixel loss.

[0142] See also Figure 8 , Figure 8 The flowchart of the method for reconstructing a three-dimensional face provided in another embodiment of the present application is shown in FIG. Figure 8 The process shown in the figure is described in detail, and the three-dimensional face reconstruction method may specifically include the following steps:

[0143] Step S701: Acquire a first training data set, wherein the first training data set includes a plurality of shape parameters and a plurality of three-dimensional face shapes, and the plurality of shape parameters and the plurality of three-dimensional face shapes correspond one to one.

[0144] In this embodiment, a first training data set is first obtained. The first training data set may include multiple shape parameters and multiple three-dimensional face shapes, and the multiple shape parameters and multiple three-dimensional face shapes correspond to each other. As a method, the first training data set may also include multiple shape parameters and multiple face coordinate maps, and the multiple shape parameters and multiple face coordinate maps correspond to each other. Figure 1 One to one correspondence.

[0145] In some embodiments, the first training data set may be stored locally on the electronic device, may be stored on other devices and sent to the electronic device, may be stored from a server and sent to the electronic device, may be collected in real time by the electronic device, etc., without limitation herein.

[0146] Step S702: using the multiple shape parameters as input parameters and the multiple three-dimensional face shapes as output parameters, training a generative adversarial network to obtain the trained first generative adversarial network.

[0147] As a method, after obtaining multiple shape parameters and multiple three-dimensional face shapes, the multiple shape parameters and multiple three-dimensional face shapes can be used as the first training data set to train the generative adversarial network to obtain a trained first generative adversarial network. In some embodiments, multiple shape parameters can be used as input parameters, and multiple three-dimensional face shapes can be used as output parameters to train the generative adversarial network to obtain a trained first generative adversarial network. In addition, after obtaining the trained first generative adversarial network, the accuracy of the trained first generative adversarial network can also be verified, and it can be determined whether the three-dimensional face shape output by the trained first generative adversarial network based on the input shape parameters meets the preset requirements. When the three-dimensional face shape output by the trained first generative adversarial network based on the input shape parameters does not meet the preset requirements, the first training data set can be re-collected to train the generative adversarial network, or multiple first training data sets can be obtained to correct the trained first generative adversarial network, which is not limited here.

[0148] Among them, in the training process of the first generative adversarial network, first, a three-dimensional face shape (face coordinate map) with the lowest resolution is trained, and the generator and discriminator are trained to be stable at this resolution. Then, transition to images with a higher resolution for training. Add a new convolution layer to the above steps, and process the new convolution layer in the form of a residual block: directly upsample the low-resolution image feature map by 2 times to an image feature map with a higher resolution, and use 1×1 convolution to convert the feature map into an RGB map F1; use 3×3 convolution to process the high-resolution image feature map that has just been upsampled by 2 times, and then use a new 1×1 convolution to convert the feature map into an RGB map F2; multiply F1 by the weight 1-a and F2 by a and add them, where the weight coefficient a gradually transitions from 0 to 1, which ensures the "fading in" of the 3×3 convolution layer, and at this resolution, the generator and discriminator are trained to be stable. Finally, transition to images with a higher resolution for training. Add new residual blocks according to the method in the above steps to process the new convolution layer, and repeat this process to finally obtain the original resolution of the three-dimensional face shape (face coordinate map).

[0149] Step S703: Acquire a second training data set, wherein the second training data set includes a plurality of texture parameters and a plurality of face texture maps, the plurality of texture parameters and the plurality of face texture maps Figure 1 One to one correspondence.

[0150] In this embodiment, a second training data set is first obtained. The second training data set may include multiple texture parameters and multiple face texture maps. Figure 1 One to one correspondence.

[0151] In some embodiments, the second training data set may be stored locally on the electronic device, may be stored on other devices and sent to the electronic device, may be stored from a server and sent to the electronic device, may be collected in real time by the electronic device, etc., without limitation herein.

[0152] In some implementations, the face coordinate map of the sample face can be obtained by performing face registration on the sample face, and added to the first training data set; the face texture map of the sample face can be obtained and added to the second training data set. As a method, the process of performing face registration on the sample face can include the following steps:

[0153] S1. Obtain a sample face, wherein the sample face is a textured three-dimensional face, project the textured three-dimensional face scan data onto a two-dimensional image to generate a frontal face image, and use a two-dimensional key point detector to generate 68 two-dimensional face key points of the frontal face image. Back-project 68 2D facial key points to generate 68 3D facial key points

[0154] S2. According to S1, 68 3D facial key points of the original 3D face scan data are obtained, and these 68 3D facial key points are aligned with the 68 3D facial key points of the standard face template through Procrustes transformation. In this way, the posture and size of all the original 3D face scan data are aligned with the standard face template.

[0155] S3. Use the NICP algorithm to align the neutral face data in the original 3D face scan data with the standard face template, where the neutral face data refers to a natural face without any expression.

[0156] S4. For the other 19 (or other numbers) facial data with expressions in the original 3D facial scan data, use a deformation transfer algorithm to transfer the expressions of a set of template faces to the registered neutral faces so that they produce corresponding expressions (for example: if the neutral expression and open mouth expression of the template face are known, the mouth opening action can be transferred to the registered neutral face).

[0157] S5. Use the NICP algorithm to align the expression data in the original 3D face scan data with the deformed, expression face template in S4, so as to generate a more accurate expression face mesh.

[0158] S6. Using the sample-based face manipulation algorithm, the 20 (or other numbers) facial meshes with expressions for each person generated in S3 and S5 are constructed into a mixed shape model of the person. The result is a neutral expression of a face and 46 FACS mixed shapes, that is, any expression H of a person can be expressed by a linear combination of mixed shapes:

[0159] S7. The parameter vector a of the mixed shape is sampled from the Gaussian distribution N (μ=0, σ=1), and then normalized by an e exponential normalization function, thereby augmenting the three-dimensional face data. The augmented face data is converted into a face coordinate map (which can be converted into a three-dimensional face shape) and a face texture map.

[0160] Step S704: using the multiple texture parameters as input parameters and the multiple face texture maps as output parameters, training the generative adversarial network to obtain the trained second generative adversarial network.

[0161] As a method, after obtaining multiple texture parameters and multiple face texture maps, the multiple texture parameters and multiple face texture maps can be used as a second training data set to train the generative adversarial network to obtain a trained second generative adversarial network. In some embodiments, multiple texture parameters can be used as input parameters, and multiple three-dimensional face texture maps can be used as output parameters to train the generative adversarial network to obtain a trained second generative adversarial network. In addition, after obtaining the trained second generative adversarial network, the accuracy of the trained second generative adversarial network can also be verified, and it can be determined whether the face coordinate map output by the trained second generative adversarial network based on the input texture parameters meets the preset requirements. When the face coordinate map output by the trained second generative adversarial network based on the input texture parameters does not meet the preset requirements, the second training data set can be re-collected to train the generative adversarial network, or multiple second training data sets can be obtained to correct the trained second generative adversarial network, which is not limited here.

[0162] Among them, in the training process of the second generative adversarial network, first, a face texture map with the lowest resolution is trained, and the generator and discriminator are trained to be stable at this resolution. Then, transition to an image with a higher resolution for training. Add a new convolution layer in the above steps, and process the new convolution layer in the form of a residual block: directly upsample the low-resolution image feature map by 2 times to an image feature map with a higher resolution, and use 1×1 convolution to convert the feature map into an RGB map F1; after processing the image feature map with a higher resolution that has just been upsampled by 2 times by 3×3 convolution, use a new 1×1 convolution to convert the feature map into an RGB map F2; multiply F1 by weight 1-a and F2 by a and add them, where the weight coefficient a gradually transitions from 0 to 1, which ensures the "fading in" of the 3×3 convolution layer, and at this resolution, the generator and discriminator are trained to be stable. Finally, transition to an image with a higher resolution for training. Add a new residual block in the above steps to process the new convolution layer, and repeat this process, and finally the original resolution of the texture coordinate map.

[0163] Step S705: Obtain shape parameters and texture parameters of the face to be reconstructed.

[0164] Step S706: input the shape parameters into the first model to obtain the three-dimensional face shape output by the first model.

[0165] Step S707: Input the texture parameters into a second model to obtain a face texture map output by the second model, wherein at least one of the first model and the second model is obtained based on generative adversarial network training.

[0166] Step S708: generating a target three-dimensional face based on the three-dimensional face shape and the face texture map, wherein the target three-dimensional face includes texture information generated based on the face texture map.

[0167] For the detailed description of steps S705 to S708 , please refer to steps S101 to S104 , which will not be repeated here.

[0168] Yet another embodiment of the present application provides a method for reconstructing a three-dimensional face. This embodiment also trains a generative adversarial network through multiple shape parameters and multiple three-dimensional face shapes to obtain a trained first generative adversarial network that can generate three-dimensional face shapes based on the shape parameters. The generative adversarial network is trained through multiple texture parameters and multiple face texture maps to obtain a trained second generative adversarial network that can generate face texture maps based on the texture parameters, so as to improve the accuracy of the generated three-dimensional face shapes and face texture maps.

[0169] See also Fig. 9 , Fig. 9 The module block diagram of the three-dimensional face reconstruction device provided by the embodiment of the present application is shown. Fig. 9 The block diagram shown in the figure is used to illustrate that the three-dimensional face reconstruction device 200 includes: a parameter acquisition module 210, a face shape acquisition module 220, a texture map acquisition module 230 and a three-dimensional face generation module 240, wherein:

[0170] The parameter acquisition module 210 is used to acquire shape parameters and texture parameters of the face to be reconstructed.

[0171] The face shape obtaining module 220 is used to input the shape parameters into the first model to obtain the three-dimensional face shape output by the first model.

[0172] Furthermore, the first model is a trained generative adversarial network, and the face shape acquisition module 220 includes: a face coordinate map acquisition submodule and a face shape acquisition submodule, wherein:

[0173] The face coordinate map obtaining submodule is used to input the shape parameters into the trained generative adversarial network to obtain the face coordinate map output by the trained generative adversarial network.

[0174] The face shape obtaining submodule is used to obtain the three-dimensional face shape based on the face coordinate map.

[0175] The texture map acquisition module 230 is used to input the texture parameters into the second model to obtain the face texture map output by the second model, wherein at least one of the first model and the second model is obtained based on generative adversarial network training.

[0176] The three-dimensional face generation module 240 is used to generate a target three-dimensional face based on the three-dimensional face shape and the face texture map, wherein the target three-dimensional face includes texture information generated based on the face texture map.

[0177] Furthermore, the 3D face reconstruction device 200 further includes: a 2D rendered face generation module, wherein:

[0178] The two-dimensional rendering face generation module is used to render the target three-dimensional face based on the rendering function, camera parameters and lighting parameters to generate a two-dimensional rendering face.

[0179] Furthermore, the three-dimensional face reconstruction device 200 further includes: a posture acquisition module, a key point loss calculation module, a first iterative optimization module and a first optimization parameter replacement module, wherein:

[0180] The posture acquisition module is used to acquire the posture of the face to be reconstructed.

[0181] Furthermore, the posture acquisition module includes: a first face key point acquisition submodule and a posture acquisition submodule, wherein:

[0182] The first facial key point acquisition submodule is used to perform key point detection on the face to be reconstructed, and obtain a plurality of three-dimensional facial key points of the face to be reconstructed as first facial key points.

[0183] Furthermore, the first facial key point acquisition submodule includes: a second facial key point acquisition unit, wherein:

[0184] The second facial key point acquisition unit is used to perform key point detection on the face to be reconstructed, obtain multiple three-dimensional facial key points of the face to be reconstructed as first facial key points, and obtain multiple two-dimensional facial key points of the face to be reconstructed as second facial key points.

[0185] The posture acquisition submodule is used to acquire the posture of the face to be reconstructed based on the first facial key points.

[0186] The key point loss calculation module is used to calculate the key point loss of the face to be reconstructed and the target three-dimensional face based on the relationship between the posture and the preset posture.

[0187] Furthermore, the key point loss calculation module includes: a first key point calculation submodule and a second key point calculation submodule, wherein:

[0188] The first key point loss calculation submodule is used to obtain multiple three-dimensional facial key points of the target three-dimensional face as third facial key points when the posture is greater than the preset posture, and calculate the key point losses of the first facial key points and the third facial key points.

[0189] The second key point calculation submodule is used to obtain multiple two-dimensional facial key points corresponding to multiple three-dimensional facial key points of the target three-dimensional face as fourth facial key points when the posture is not greater than the preset posture, and calculate the key point losses of the second facial key points and the fourth facial key points.

[0190] The first iterative optimization module is used to iteratively optimize the shape parameters based on the key point loss to obtain first optimized shape parameters.

[0191] The first optimization parameter replacement module is used to first input the first optimized shape parameter as a new shape parameter into the first model.

[0192] Furthermore, the three-dimensional face reconstruction device 200 further includes: an optimized camera parameter acquisition module and a second optimized parameter replacement module, wherein:

[0193] The optimized camera parameter acquisition module is used to iteratively optimize the camera parameters based on the key point loss to obtain optimized camera parameters.

[0194] The second optimized parameter replacement module is used to use the optimized camera parameters as new camera parameters to render the target three-dimensional human face.

[0195] Furthermore, the three-dimensional face reconstruction device 200 further includes: a biometric information acquisition module, a biometric loss calculation module, a second iterative optimization module and a third optimization parameter replacement module, wherein:

[0196] The biometric information acquisition module is used to acquire the biometric information of the face to be reconstructed as the first biometric information, and acquire the biometric information of the two-dimensionally rendered face as the second biometric information.

[0197] A biometric loss calculation module is used to calculate the biometric loss of the face to be reconstructed and the two-dimensional rendered face based on the first biometric information and the second biometric information.

[0198] The second iterative optimization module is used to iteratively optimize the shape parameters and the texture parameters based on the biometric loss to obtain second optimized shape parameters and first optimized texture parameters.

[0199] The third optimization parameter replacement module is used to input the second optimized shape parameter as a new shape parameter into the first model, and input the first optimized texture parameter as a new texture parameter into the second model.

[0200] Furthermore, the three-dimensional face reconstruction device 200 further includes: an attribute content information acquisition module, an attribute content loss calculation module, a third iterative optimization module and a fourth optimization parameter replacement module, wherein:

[0201] The attribute content information acquisition module is used to acquire the attribute content information of the to-be-reconstructed face as the first attribute content information, and acquire the attribute content information of the two-dimensionally rendered face as the second attribute content information.

[0202] The attribute content loss calculation module is used to calculate the attribute content loss of the to-be-reconstructed face and the two-dimensionally rendered face based on the first attribute content information and the second attribute content information.

[0203] The third iterative optimization module is used to iteratively optimize the shape parameters and the texture parameters based on the attribute content loss to obtain third optimized shape parameters and second optimized texture parameters.

[0204] The fourth optimization parameter replacement module is used to input the third optimized shape parameter as a new shape parameter into the first model, and input the second optimized texture parameter as a new texture parameter into the second model.

[0205] Furthermore, the three-dimensional face reconstruction device 200 further includes: a pixel information acquisition module, a pixel loss calculation module, a fourth iterative optimization module and a fifth optimization parameter replacement module, wherein:

[0206] The pixel information acquisition module is used to acquire pixel information of the to-be-reconstructed face as first pixel information, and acquire pixel information based on the two-dimensionally rendered face as second pixel information.

[0207] A pixel loss calculation module is used to calculate the pixel loss of the to-be-reconstructed face and the two-dimensionally rendered face based on the first pixel information and the second pixel information.

[0208] The fourth iterative optimization module is used to iteratively optimize the illumination parameters based on the pixel loss to obtain optimized illumination parameters.

[0209] The fifth optimization parameter replacement module is used to use the optimized illumination parameters as new illumination parameters to render the target three-dimensional human face.

[0210] Furthermore, the three-dimensional face reconstruction device 200 further includes: a first training data set acquisition module and a first generative adversarial network acquisition module, wherein:

[0211] The first training data set acquisition module is used to acquire a first training data set, wherein the first training data set includes multiple shape parameters and multiple three-dimensional face shapes, and the multiple shape parameters and the multiple three-dimensional face shapes correspond to each other one by one.

[0212] The first generative adversarial network acquisition module is used to train the generative adversarial network by taking the multiple shape parameters as input parameters and the multiple three-dimensional face shapes as output parameters to obtain the trained first generative adversarial network.

[0213] Furthermore, the three-dimensional face reconstruction device 200 further includes: a second training data set acquisition module and a second generative adversarial network acquisition module, wherein:

[0214] The second training data set acquisition module is used to acquire a second training data set, wherein the second training data set includes a plurality of texture parameters and a plurality of face texture maps, the plurality of texture parameters and the plurality of face texture maps Figure 1 One to one correspondence.

[0215] The second generative adversarial network acquisition module is used to train the generative adversarial network by taking the multiple texture parameters as input parameters and the multiple face texture maps as output parameters to obtain the trained second generative adversarial network.

[0216] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0217] In several embodiments provided in the present application, the coupling between modules may be electrical, mechanical or other forms of coupling.

[0218] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0219] See also Fig.10 , which shows a structural block diagram of an electronic device 100 provided in an embodiment of the present application. The electronic device 100 may be an electronic device capable of running applications, such as a smart phone, a tablet computer, an e-book, etc. The electronic device 100 in the present application may include one or more of the following components: a processor 110, a memory 120, a touch screen 130, and one or more applications, wherein one or more applications may be stored in the memory 120 and configured to be executed by one or more processors 110, and one or more programs are configured to execute the method described in the aforementioned method embodiment.

[0220] Among them, the processor 110 may include one or more processing cores. The processor 110 uses various interfaces and lines to connect various parts of the entire electronic device 100, and executes various functions and processes data of the electronic device 100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 120, and calling data stored in the memory 120. Optionally, the processor 110 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 110 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 110, but may be implemented separately through a communication chip.

[0221] The memory 120 may include a random access memory (RAM) or a read-only memory (ROM). The memory 120 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as a phone book, audio and video data, chat record data) created by the terminal 100 during use.

[0222] The touch screen 130 is used to display information input by the user, information provided to the user, and various graphical user interfaces of the electronic device 100. These graphical user interfaces can be composed of graphics, text, icons, numbers, videos, and any combination thereof. In one example, the touch screen 130 can be a liquid crystal display (LCD) or an organic light-emitting diode (OLED), which is not limited here.

[0223] See also Fig.11 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 300 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0224] The computer readable storage medium 300 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 300 includes a non-transitory computer-readable storage medium. The computer readable storage medium 300 has storage space for program code 310 that performs any method steps in the above method. These program codes can be read from or written to one or more computer program products. The program code 310 can be compressed, for example, in an appropriate form.

[0225] In summary, the three-dimensional face reconstruction method, device, electronic device and storage medium provided in the embodiments of the present application obtain shape parameters and texture parameters of the face to be reconstructed, input the shape parameters into the first model, obtain the three-dimensional face shape output by the first model, input the texture parameters into the second model, obtain the face texture map output by the second model, wherein at least one of the first model and the second model is obtained based on generative adversarial network training, and based on the three-dimensional face shape and the face texture map, a target three-dimensional face is generated, wherein the target three-dimensional face includes texture information generated based on the face texture map, so that the three-dimensional face shape and / or the face texture map are generated by the trained generative adversarial network, so that the generated target three-dimensional face has rich detail features, thereby improving the reconstruction effect of the three-dimensional face.

[0226] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for reconstructing a three-dimensional face, characterized in that: The method comprises: Obtaining shape parameters and texture parameters of the face to be reconstructed; Inputting the shape parameters into a trained first generative adversarial network to obtain a three-dimensional face shape output by the first generative adversarial network; Inputting the texture parameters into a trained second generative adversarial network to obtain a face texture map output by the second generative adversarial network; Based on the three-dimensional face shape and the face texture map, generating a target three-dimensional face, wherein the target three-dimensional face includes texture information generated based on the face texture map; Rendering the target three-dimensional face based on a rendering function, camera parameters, and lighting parameters to generate a two-dimensional rendered face; Performing key point detection on the face to be reconstructed, obtaining a plurality of three-dimensional face key points of the face to be reconstructed as first face key points, and obtaining a plurality of two-dimensional face key points of the face to be reconstructed as second face key points; Based on the first facial key points, acquiring the posture of the face to be reconstructed; When the posture is greater than a preset posture, obtaining a plurality of three-dimensional facial key points of the target three-dimensional face as third facial key points, and calculating key point losses of the first facial key points and the third facial key points; When the posture is not greater than the preset posture, obtaining a plurality of two-dimensional face key points corresponding to a plurality of three-dimensional face key points of the target three-dimensional face as fourth face key points, and calculating the key point loss of the second face key points and the fourth face key points; Iteratively optimize the shape parameters based on the key point loss to obtain first optimized shape parameters; The first optimized shape parameter is input into the first generative adversarial network as a new shape parameter.

2. The method according to claim 1, characterized in that The step of inputting the shape parameters into a trained first generative adversarial network to obtain a three-dimensional face shape output by the first generative adversarial network includes: Inputting the shape parameters into the first generative adversarial network to obtain a face coordinate map output by the first generative adversarial network; Based on the face coordinate map, the three-dimensional face shape is obtained.

3. The method according to claim 1, characterized in that The method further comprises: Iteratively optimize the camera parameters based on the key point loss to obtain optimized camera parameters; The optimized camera parameters are used as new camera parameters to render the target three-dimensional human face.

4. The method according to claim 1, characterized in that: After rendering the target three-dimensional face based on the rendering function, the camera parameters and the illumination parameters to generate a two-dimensional rendered face, the method further includes: Acquire biometric information of the face to be reconstructed as first biometric information, and acquire biometric information of the two-dimensionally rendered face as second biometric information; Calculating the biometric loss of the to-be-reconstructed face and the two-dimensionally rendered face based on the first biometric information and the second biometric information; Iteratively optimize the shape parameter and the texture parameter based on the biometric loss to obtain a second optimized shape parameter and a first optimized texture parameter; The second optimized shape parameter is input into the first generative adversarial network as a new shape parameter, and the first optimized texture parameter is input into the second generative adversarial network as a new texture parameter.

5. The method according to claim 1, characterized in that After rendering the target three-dimensional face based on the rendering function, the camera parameters and the illumination parameters to generate a two-dimensional rendered face, the method further includes: Acquiring attribute content information of the to-be-reconstructed face as first attribute content information, and acquiring attribute content information of the two-dimensionally rendered face as second attribute content information; Calculating attribute content losses of the to-be-reconstructed face and the two-dimensionally rendered face based on the first attribute content information and the second attribute content information; Iteratively optimize the shape parameters and the texture parameters based on the attribute content loss to obtain third optimized shape parameters and second optimized texture parameters; The third optimized shape parameter is input into the first generative adversarial network as a new shape parameter, and the second optimized texture parameter is input into the second generative adversarial network as a new texture parameter.

6. The method according to claim 1, characterized in that After rendering the target three-dimensional face based on the rendering function, the camera parameters and the illumination parameters to generate a two-dimensional rendered face, the method further includes: Acquire pixel information of the to-be-reconstructed face as first pixel information, and acquire pixel information based on the two-dimensionally rendered face as second pixel information; Calculating pixel losses of the to-be-reconstructed face and the two-dimensionally rendered face based on the first pixel information and the second pixel information; Iteratively optimizing the illumination parameters based on the pixel loss to obtain optimized illumination parameters; The optimized illumination parameters are used as new illumination parameters to render the target three-dimensional human face.

7. The method according to any one of claims 1 to 6, characterized in that: Before inputting the shape parameters into the trained first generative adversarial network to obtain the three-dimensional face shape output by the first generative adversarial network, the method further includes: Acquire a first training data set, wherein the first training data set includes a plurality of shape parameters and a plurality of three-dimensional face shapes, and the plurality of shape parameters and the plurality of three-dimensional face shapes correspond to each other one by one; The multiple shape parameters are used as input parameters and the multiple three-dimensional face shapes are used as output parameters to train a generative adversarial network to obtain the trained first generative adversarial network.

8. The method according to any one of claims 1 to 6, characterized in that: Before inputting the texture parameters into the trained second generative adversarial network to obtain the face texture map output by the second generative adversarial network, the method further includes: Acquire a second training data set, wherein the second training data set includes a plurality of texture parameters and a plurality of face texture maps, and the plurality of texture parameters and the plurality of face texture maps correspond one to one; The plurality of texture parameters are used as input parameters and the plurality of face texture maps are used as output parameters to train a generative adversarial network to obtain the trained second generative adversarial network.

9. A three-dimensional face reconstruction device, characterized in that: The device comprises: A parameter acquisition module, used to obtain shape parameters and texture parameters of the face to be reconstructed; A face shape acquisition module, used for inputting the shape parameters into a trained first generative adversarial network to obtain a three-dimensional face shape output by the first generative adversarial network; A texture map acquisition module, used for inputting the texture parameters into a trained second generative adversarial network to obtain a face texture map output by the second generative adversarial network; A three-dimensional face generation module, used to generate a target three-dimensional face based on the three-dimensional face shape and the face texture map, wherein the target three-dimensional face includes texture information generated based on the face texture map; A two-dimensional rendering face generation module, used for rendering the target three-dimensional face based on a rendering function, camera parameters and illumination parameters to generate a two-dimensional rendering face; a posture acquisition module, used to perform key point detection on the face to be reconstructed, obtain a plurality of three-dimensional face key points of the face to be reconstructed as first face key points, and obtain a plurality of two-dimensional face key points of the face to be reconstructed as second face key points; A posture acquisition module is further used to acquire the posture of the face to be reconstructed based on the first facial key points; a key point loss calculation module, configured to obtain a plurality of three-dimensional face key points of the target three-dimensional face as third face key points when the posture is greater than a preset posture, and calculate key point losses of the first face key points and the third face key points; The key point loss calculation module is further used to obtain multiple two-dimensional face key points corresponding to multiple three-dimensional face key points of the target three-dimensional face as fourth face key points when the posture is not greater than the preset posture, and calculate the key point loss of the second face key points and the fourth face key points; A first iterative optimization module, configured to iteratively optimize the shape parameters based on the key point loss to obtain first optimized shape parameters; The first optimized parameter replacement module is used to input the first optimized shape parameter as a new shape parameter into the first generative adversarial network.

10. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory is coupled to the processor, and the memory stores instructions. When the instructions are executed by the processor, the processor executes the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 8.

Images