Face reconstruction method, device, electronic device, and storage medium

Abstract

The present disclosure provides a method for face reconstruction, which relates to the field of computer vision technology, and in particular to a face reconstruction device, an electronic device, and a storage medium. The specific implementation scheme is: generating a face point cloud model based on face images of multiple perspectives of a target face; determining the model transformation parameters of an initial three-dimensional face model based on the face image and the face point cloud model, and transforming the initial three-dimensional face model based on the model transformation parameters to obtain a three-dimensional face model corresponding to the target face.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to a method, device, electronic device, and storage medium for face reconstruction. Background Art

[0002] At present, face reconstruction technology is widely used in the field of pan-entertainment, including face video special effects, virtual human live broadcast, game modeling, movies, etc. At present, there are many related technical solutions for face reconstruction, such as: using a single-view face image for face reconstruction, but this method lacks depth information, and the reconstructed face has poor three-dimensional structure accuracy and stiff expression. Summary of the invention

[0003] The present disclosure provides a method, device, electronic device, and storage medium for face reconstruction to solve at least one of the above-mentioned technical problems.

[0004] According to one aspect of the present disclosure, a method for face reconstruction is provided, the method comprising:

[0005] Generate a face point cloud model based on face images of the target face from multiple perspectives;

[0006] Based on the face image and the face point cloud model, model transformation parameters of an initial three-dimensional face model are determined, and based on the model transformation parameters, the initial three-dimensional face model is transformed to obtain a three-dimensional face model corresponding to the target face.

[0007] According to another aspect of the present disclosure, a device for face reconstruction is provided, the device comprising:

[0008] A face point cloud model generation module is used to generate a face point cloud model based on face images of a target face from multiple perspectives;

[0009] The 3D face model generation module is used to determine the model transformation parameters of the initial 3D face model based on the face image and the face point cloud model, and transform the initial 3D face model based on the model transformation parameters to obtain a 3D face model corresponding to the target face.

[0010] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0011] at least one processor; and

[0012] a memory communicatively connected to the at least one processor; wherein,

[0013] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above method.

[0014] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the above method.

[0015] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the above method when executed by a processor.

[0016] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0018] Figure 1 is a flowchart of a method for face reconstruction provided by the first embodiment of the present disclosure;

[0019] Figure 2 is a flowchart of a method for face reconstruction provided by the second embodiment of the present disclosure;

[0020] Figure 3 is a flowchart of S205 of a face reconstruction method provided by the second embodiment of the present disclosure;

[0021] Figure 4 is a flowchart of S206 of a face reconstruction method provided by the second embodiment of the present disclosure;

[0022] Figure 5 is a structural schematic diagram of a face reconstruction device provided by the third embodiment of the present disclosure;

[0023] Figure 6 The block diagram is a block diagram of an electronic device for implementing the method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0025] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0026] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0027] The terminology used herein is for describing particular embodiments only and is not intended to limit the present disclosure.As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0028] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined as such herein.

[0029] The method for copying a folder of vehicle safety assets according to the present disclosure can be executed by an electronic device such as a terminal device or a server, and the terminal device can be a vehicle-mounted device, a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The method can be implemented by a processor calling a computer-readable program instruction stored in a memory. Alternatively, the face reconstruction method provided by the present disclosure can be executed by a server.

[0030] In the first embodiment disclosed, see Figure 1 , Figure 1 A schematic flow chart of a method for face reconstruction provided by the first embodiment of the present disclosure is shown. The method comprises:

[0031] S101, generating a facial point cloud model based on facial images of a target face from multiple perspectives.

[0032] S102: determining model transformation parameters of an initial three-dimensional face model based on the face image and the face point cloud model, and transforming the initial three-dimensional face model based on the model transformation parameters to obtain a three-dimensional face model corresponding to the target face.

[0033] It should be noted that the face reconstruction method provided by the present disclosure can be applied in a variety of application scenarios.

[0034] For example, when the face reconstruction method provided by the present invention is applied to a game, an image including the face of a game player can be acquired through an image acquisition device installed in the game device, or an image including the face of the game player can be selected from an album in the game device and the acquired image including the face of the game player can be used as the target face image.

[0035] For another example, when the face reconstruction method provided by the present invention is applied to terminal devices such as mobile phones, the camera of the terminal device can capture an image including the user's face, or an image including the user's face can be selected from the photo album of the terminal device, or a face image including the user's face can be received from other applications installed in the terminal device.

[0036] For another example, when the face reconstruction method provided by the present disclosure is applied to a live broadcast scenario, a video frame image containing a face can be obtained from multiple video frame images included in a video stream obtained by a live broadcast device; and the video frame image containing the face is used as a target face image. Here, the target face image can, for example, have multiple frames; and the multiple target face images can, for example, be obtained by sampling multiple video frame images in a video stream.

[0037] For another example, when the face reconstruction method provided by the present disclosure is applied to a special effects scene in a movie or animation, a depth photography device can be used to capture a color depth face image of the actor.

[0038] In some examples, in S101, the face image may be multiple types of images, such as: a two-dimensional face image, a color depth image, a high-precision color depth image, etc.

[0039] Among them, the color depth image is also called red green blue depth (Red Green Blue Depth, RGBD) image.

[0040] In some examples, in S101, the face point cloud model generated based on the face image is a three-dimensional model composed of point clouds, and the point cloud is a data set, and each point cloud data in the data set includes at least coordinates (X, Y, Z three dimensions), and may also include intensity values, color values, classification values, and time light information, wherein the intensity value records the intensity of the return signal according to the reflectivity of the object surface (face surface). Multiple point cloud data are combined to form a face point cloud model, that is, a collection of data points in space that represent a three-dimensional shape or object (that is, a three-dimensional face object in this disclosure).

[0041] In some examples, in S102, a 3D face model can be reconstructed based on the face image and the face point cloud model in a variety of ways, such as: stereo matching (Structure From Motion, SfM) technology, shadow modeling (Shape from Shading, SFS) technology, 3D deformable face (3DMM) model, etc. For ease of explanation, the initial 3D face model in the present disclosure is a 3DMM model, and the 3D face model is a face model obtained after optimization based on the 3DMM model, but this does not constitute a limitation to the present disclosure.

[0042] The model transformation parameters include various types of parameters for performing deformation transformation and pixel transformation on the model, for example, shape basis parameters of a 3D face model, pixel information of pixel points of a rendering of a 3D face model, and the like.

[0043] The face point cloud model has high-precision features and details close to the real face. However, the face point cloud model generated by face images from multiple perspectives is unstable. For example, the point cloud may be sparse or missing in some areas due to the quality of the collected images. For another example, the topological structure of the face point cloud model generated by the face image is not fixed, that is, the number of points in the face point cloud generated by each image is uncertain and has no semantic information. Therefore, the face point cloud model cannot be used directly for driving. Based on this, it is necessary to convert the face point cloud model generated by S101 into a three-dimensional face model (such as a 3DMM model) with a stable topological structure in S102, and further optimize the three-dimensional face model based on the two-dimensional face image.

[0044] In the present disclosure, a three-dimensional face model is three-dimensionally constrained by a high-precision face point cloud model, and a three-dimensional face model is two-dimensionally constrained by a face image with semantic information, so that a high-precision and high-detail three-dimensional face model can be reconstructed.

[0045] In disclosing the second embodiment, see Figure 2 , Figure 2 A flow chart of a method for face reconstruction provided by the second embodiment of the present disclosure is shown. The method comprises:

[0046] S201, obtaining camera parameters.

[0047] The camera parameters are camera parameters of a camera for collecting facial images from multiple perspectives. Facial images can be collected through cameras from multiple perspectives or through a single camera.

[0048] The camera parameters include camera intrinsic parameters and camera extrinsic parameters.

[0049] Specifically, the camera intrinsic parameters are parameters related to the camera's own characteristics, such as the camera's focal length, pixel size, etc.; the camera extrinsic parameters are parameters of the camera in the world coordinate system, such as the camera's position, rotation direction, etc.

[0050] Among them, the specific purpose of S201 is to perform camera calibration to achieve the mapping from world coordinates to pixel coordinates.

[0051] In some examples, the camera intrinsic parameters are pre-calibrated parameters, and only the camera extrinsic parameters need to be determined.

[0052] S202: Generate a facial point cloud model according to camera parameters and facial images from multiple perspectives.

[0053] In some examples, a multi-viewstereo (MVS) algorithm can be used to generate a facial point cloud model based on facial images from multiple perspectives. Based on the MVS algorithm, facial images from multiple dense perspectives are used as input to automatically extract feature points in the image, and the illumination consistency of the feature points at the same position between the images is calculated for matching, thereby completing face reconstruction.

[0054] Specifically, for each feature point on the face image, the pixel intensity of the feature point, the mean and variance of the pixels in the surrounding neighborhood, and other information are calculated, and these information are used as the features of the feature point. Then, the feature difference between the feature points in the two images is calculated. When the difference is less than a certain threshold, the two feature points are considered to be matched successfully. Based on this method, the internal and external parameters of the camera are determined, and the face images of multiple perspectives are spatially transformed to generate the corresponding face point cloud model.

[0055] In some examples, the Structure from Motion (SFM) method and the Bundle Adjustment (BA) method can be used to complete the alignment of face images, thereby obtaining the intrinsic and extrinsic parameters of the camera under multiple perspectives.

[0056] The SFM algorithm, without any camera calibration prior information, directly determines the two-dimensional feature point pairs at the same position of multiple matching face images through the illumination consistency between multiple images, and then preliminarily estimates the extrinsic parameters between multiple cameras based on the matching two-dimensional feature point pairs, obtains a three-dimensional point cloud, and then optimizes the camera's extrinsic parameters through the BA algorithm.

[0057] S203, projecting the face point cloud model onto face images at multiple perspectives based on camera external parameters, and projecting the face point cloud model at each perspective onto the face image corresponding to each perspective based on camera internal parameters.

[0058] S204: Determine the correspondence between each point of the face point cloud model and each pixel point on the face image.

[0059] The face point cloud model has high-precision features and details close to the real face. However, the topological structure of the face point cloud model generated by the face image is not fixed, that is, the number of points of the face point cloud generated by each image is uncertain and has no semantic information. Therefore, the camera external parameters can be used to project the various perspectives of the three-dimensional face movie model onto the face images under multiple perspectives by means of rotation, translation, etc. For the face image under each perspective, the camera internal parameters are used to project the face point cloud model corresponding to the face image under the perspective onto the face image under the perspective by means of focal length conversion, scaling, etc. In this way, the correspondence between each point of the entire face point cloud model and the pixel points on the face image under each perspective can be determined, so that any two of the face images, face point cloud models, and three-dimensional face models can be aligned through this correspondence in the future. In addition, semantic information can be added to the points on the face point cloud model, so that each point of the high-precision face point cloud model has accurate three-dimensional semantic information, so that it can be subsequently aligned with the three-dimensional face model through semantic information.

[0060] Among them, S201-S204 is an implementation method of S101. Specifically, S101 also has other implementation methods, which are not limited here.

[0061] In some examples, the present disclosure can perform two-stage optimization of the 3D face model. In the first stage, the entire 3D face model is transformed; in the second stage, each point of the 3D face model is transformed. In other words, a rough adjustment is performed first, and then a fine optimization is performed based on the roughly optimized 3D face model. In this way, a high-precision and delicate 3D face model can be obtained.

[0062] Of course, in some implementations, only the first stage of adjustment may be performed, and the second stage of adjustment may be omitted to reduce the amount of calculation and save system resources, which is not limited here.

[0063] Based on the above, for the optimization of the first stage (S205-S206), the model transformation parameters include: model overall transformation parameters, and the method provided by the present disclosure also includes:

[0064] S205: iteratively update the overall model transformation parameters of the initial three-dimensional face model based on the face image and the face point cloud model.

[0065] The overall transformation parameters of the initial three-dimensional face model are iteratively updated based on the face image and the face point cloud model. The purpose of the iterative update is to make the final three-dimensional face model as close to the face image and the face point cloud model as possible.

[0066] Specifically, see Figure 3 , Figure 3 A schematic flow chart of S205 is shown, where S205 includes:

[0067] The overall transformation parameters of the model include at least: deformation parameters.

[0068] S2051. Perform at least one round of iterative update on the deformation parameters of the initial three-dimensional face model based on the face image and the face point cloud model, and obtain the iteratively updated deformation parameters based on a preset first iteration condition.

[0069] Among them, any round of at least one round of iterative updates in S2051 includes:

[0070] S2051A. Deform the initial three-dimensional face model according to the deformation parameters of the current round to obtain a deformed three-dimensional face model.

[0071] In the first iteration, the current deformation parameters are the default deformation parameters; in the iterations after the first, the current deformation parameters are the deformation parameters updated in the previous iteration.

[0072] Among them, the initial three-dimensional face model is a 3DMM model, and the expression of the 3DMM model is:

[0073] face=mean+cs×shp+ce×exp

[0074] Among them, face is a 3D face model, mean is a preset average 3D face model (i.e., an initialized average 3D face model), shp represents the shape basis of the initial 3D face model, cs is a shape basis parameter, exp represents the expression basis of the initial 3D model, and ce is an expression basis parameter; by changing ce and cs, different 3D face models can be expressed. The overall transformation parameters of the model include at least deformation parameters, which include at least shape basis parameters and expression basis parameters; they may also include: illumination parameters, rotation parameters, translation parameters, etc.

[0075] S2051B. Determine a first error based on key feature points of the facial image and key feature points of the deformed three-dimensional facial model.

[0076] Among them, the key feature points of the three-dimensional face model can be determined in a variety of ways, for example, by extracting the key feature points through the Active Shape Model (ASM), the key feature points are, for example, the corners of the eyes, the tip of the nose, the corners of the mouth and other key feature points.

[0077] Furthermore, key feature points may be extracted from multiple perspectives, or key feature points may be extracted from the front face, that is, in S2052B, the facial image is a frontal facial image, and the three-dimensional facial model is also a three-dimensional facial model from a frontal perspective.

[0078] In some examples, S2052B includes:

[0079] Sub-step 1: Determine each key feature point pair.

[0080] A key feature point pair includes a key feature point of a deformed three-dimensional face model and a key feature point of a face image corresponding to the key feature point of the deformed three-dimensional face model.

[0081] Sub-step 2: determining a first error based on a relative distance between two key feature points of each key feature point pair in each key feature point pair.

[0082] In sub-step 1 and sub-step 2 of S2051B, the key feature points of the three-dimensional face model and the key feature points of the corresponding face image are aligned through at least one of rotation, translation, scaling, etc., and the key feature points with the same semantics and / or the same position between the two are determined as a key feature point pair, and then the relative distance between the two key feature points in the key feature point pair is calculated, and the relative distance is taken as the first error.

[0083] S2051C. Determine a second error based on each point of the facial point cloud model and each point of the deformed three-dimensional face model.

[0084] In some examples, the S2051C includes:

[0085] Sub-step 1: Determine each point pair.

[0086] Among them, a point pair includes a target point of a deformed three-dimensional face model and the closest point of the target point in the face point cloud model;

[0087] Sub-step 2: filter the nearest point of each point pair in each point pair, and update the point pair according to the filter result;

[0088] Sub-step three: determining a second error based on the relative distance between the target point and the nearest point of each point pair in each updated point pair.

[0089] In sub-step 1 to sub-step 3 of S2051C, since the correspondence between the points of the point cloud and the pixels of the face image is determined in S203-S204 above, the points of the face point cloud model have semantic information. According to the coordinates of the semantic information of the points of the face point cloud model and the semantic information and coordinates of the three-dimensional face model, the two are aligned, and for any point (target point) of the three-dimensional face model, the N points on the face point cloud model whose relative distance to the projection position of the target point on the face point cloud model is closest to the point are determined as the nearest point of the target point, and then the target point and the nearest point of the target point are determined as a point pair, where N is an integer greater than or equal to 1.

[0090] For a point pair, the overall relative distance (for example, taking the average) is determined based on the single-point relative distance between the nearest point and the target point, and the relative distance is used as the second error.

[0091] S2051D. Update the deformation parameters based on the first error and the second error.

[0092] The preset first iteration condition is that at least one of the value of the first error and the value of the second error no longer decreases.

[0093] The purpose of iteratively updating the deformation parameters based on the first error is to make the first error as small as possible, so that the key feature points in each key feature point pair are as close as possible, and further make the three-dimensional face model and the face image as close as possible.

[0094] The purpose of iteratively updating the deformation parameters based on the second error is to make the second error as small as possible, so that the points in each point pair are as close as possible, and further make the three-dimensional face model and the face point cloud model as close as possible.

[0095] The updated deformation parameters determined in this round of iteration are back-propagated to the 3D face model for deformation processing, and the first error and the second error are repeatedly determined, and the deformation parameters are updated with the first error and the second error until the first error and the second error no longer decrease, that is, the optimization can no longer be continued, and the optimal result is achieved. In this case, the 3D face model is constrained based on the face image and the face point cloud model, and it is iteratively updated so that the 3D face model after deformation processing is close to the real face, and a high-precision and delicate 3D face model is obtained.

[0096] Furthermore, in some examples, the overall model transformation parameters also include: pixel area transformation parameters.

[0097] Based on this, continue to see Figure 3 , S205 also includes:

[0098] S2052, performing at least one round of iterative updating on the pixel region transformation parameters of the initial three-dimensional face model based on the face image and the face point cloud model, and obtaining the iteratively updated pixel region transformation parameters based on a preset second iteration condition;

[0099] Among them, any round of at least one round of iterative update in S2052 includes:

[0100] S2052A: Perform pixel transformation processing on the rendering of the initial three-dimensional face model according to the pixel area transformation parameters of the current round to obtain a rendering of the three-dimensional face model after pixel transformation.

[0101] The 3D face model (including the initial 3D face model) includes not only a shape base and an expression base, but also a face map (also called a rendering) after rendering the 3D face model, so that the texture of the 3D face model is close to that of a real face.

[0102] In the first iteration, the current pixel region transformation parameters are the default pixel region transformation parameters; in the iterations after the first, the current pixel region transformation parameters are the pixel region transformation parameters updated in the previous iteration.

[0103] S2052B, determining a third error based on pixel information of each pixel point of the face image and pixel information of each pixel point of the rendering of the three-dimensional face model after pixel transformation.

[0104] In some examples, S2052B includes:

[0105] Sub-step 1: determining each face region group, wherein a face region group includes a first face region of a rendering of a three-dimensional face model after pixel transformation and a second face region of a face image corresponding to the first face region;

[0106] Sub-step 2: determining first area pixels of the first face area and second area pixels of the second face area of ​​each face area pair in each face area group;

[0107] Sub-step three: determining a third error based on the difference between the first region pixels and the second region pixels of each face region in each face region group.

[0108] In sub-step 1 to sub-step 3 of S2052B, the three-dimensional face model is first divided into multiple first face areas (for example, hair, face, eyes, etc.), and the face image is divided into multiple second face areas in the same way; after the three-dimensional face model and the face image are aligned based on key feature points, the first face areas and the second face areas with the same semantics and / or the same position are determined as a face area group; for a first face area or a second face area, based on the pixel information of multiple pixels contained in the first face area or the pixel information of multiple pixels contained in the second face area, the first area pixels (for example, the mean of the pixel information of the pixels) or the second area pixels (for example, the mean of the pixel information of the pixels) are determined, and then based on the difference between the first area pixels and the second area pixels of the multiple face area groups, the third error of the overall face area is determined, and the essence of the third error can be: pixel color error, pixel brightness error, etc.

[0109] Accordingly, the pixel information includes at least pixel color, pixel brightness, etc.

[0110] S2052C. Update pixel region transformation parameters based on the third error.

[0111] The preset second iteration condition is that the value of the third error does not decrease any more.

[0112] The purpose of iteratively updating the pixel area transformation parameters based on the third error is to make the third error as small as possible, so that the first area pixels and the second area pixels in each face area group are as close as possible (that is, the color or brightness is as similar as possible), thereby making the rendering of the three-dimensional face model and the face image as close as possible.

[0113] The essence of the pixel region transformation parameter may be: the color value and brightness value of each pixel point in a first face region.

[0114] The updated pixel area transformation parameters determined in this round of iteration are back-propagated to the 3D face model for pixel transformation processing, and the third error is determined repeatedly, and the pixel area transformation parameters are updated with the third error until the third error no longer decreases, that is, optimization can no longer be continued, and the optimal result is achieved. In this case, not only the shape of the 3D face model is constrained based on the face image and the face point cloud model, but also the rendering effect of the 3D face model is constrained based on the rendering image and face image of the 3D face model, so that the texture of the 3D face model after deformation processing is close to the real face, and a high-precision and delicate 3D face model is obtained.

[0115] S206: transform the initial three-dimensional face model according to the iteratively updated overall model transformation parameters to obtain a three-dimensional face model.

[0116] The first stage of optimization is to optimize the overall transformation parameters of the model. The optimization target is the deformation parameters of the model (such as ce, cs) and the pixel area transformation parameters of the face area, so that the shape and texture of the three-dimensional face model are close to the face image and face point cloud model.

[0117] After the optimization of the first stage, that is, after S206, the method provided by the present disclosure may further include optimization for the second stage (S207-S208), wherein the model transformation parameters include: model point transformation parameters:

[0118] S207: iteratively update the model point transformation parameters of each point of the three-dimensional face model based on the face image and the face point cloud model.

[0119] In the second stage of optimization, each point of the three-dimensional face model is used as a variable for optimization, and the three-dimensional face model is optimized based on each point to adjust the three-dimensional face model with higher accuracy.

[0120] In particular, in some examples, see Figure 4 , Figure 4 A schematic flow chart of S207 is shown, where S207 includes:

[0121] The model point transformation parameters include at least three-dimensional vertex coordinates.

[0122] S2071. Based on each key feature point of the face image, iteratively update the three-dimensional vertex coordinates of each key feature point of the three-dimensional face model, and obtain the three-dimensional vertex coordinates of each key feature point after iterative update based on a preset third iteration condition.

[0123] Among them, any round of at least one round of iterative update in S2071 includes:

[0124] S2071A. Transform the three-dimensional face model according to the three-dimensional vertex coordinates corresponding to the key feature points of each three-dimensional face model in the current round to obtain a transformed three-dimensional face model.

[0125] In the first iteration, the current three-dimensional vertex coordinates are the three-dimensional vertex coordinates of the three-dimensional face model optimized based on S206; in iterations after the first, the current three-dimensional vertex coordinates are the three-dimensional vertex coordinates updated in the previous iteration.

[0126] S2071B. For any key feature point among the key feature points, determine a fourth error based on a relative distance between the key feature point of the face image and the key feature point of the transformed three-dimensional face model.

[0127] Among them, the key feature points of the three-dimensional face model can be determined in a variety of ways, for example, by extracting the key feature points through the Active Shape Model (ASM), such as the corners of the eyes, the tip of the nose, the corners of the mouth and other key feature points.

[0128] Furthermore, key feature points may be extracted from multiple perspectives, or key feature points may be extracted from the front face, that is, in S2071B, the facial image is a frontal facial image, and the three-dimensional facial model is also a three-dimensional facial model from a frontal perspective.

[0129] In some examples, S2071B includes:

[0130] Sub-step 1: Determine each key feature point pair.

[0131] A key feature point pair includes a key feature point of a deformed three-dimensional face model and a key feature point of a face image corresponding to the key feature point of the deformed three-dimensional face model.

[0132] Sub-step 2: for any key feature point pair, based on the relative distance between the key feature point of the face image and the key feature point of the transformed three-dimensional face model, determine the fourth error for the key feature point.

[0133] In sub-step 1 and sub-step 2 of S2071B, the key feature points of the three-dimensional face model and the key feature points of the corresponding face image are aligned through at least one of rotation, translation, scaling, etc., and the key feature points with the same semantics and / or the same position between the two are determined as a key feature point pair, and then the relative distance between the two key feature points in each key feature point pair is calculated, and the relative distance is taken as the fourth error of a key feature point.

[0134] S2071C. Update the three-dimensional vertex coordinates corresponding to the key feature point based on the fourth error.

[0135] For any key feature point of a three-dimensional face model, based on the fourth error of the key feature point, the three-dimensional vertex coordinates of the key feature point are optimized so that the key feature point of the three-dimensional face model is as close as possible to the key feature point of the corresponding face image.

[0136] The third iteration condition is that the value of the fourth error does not decrease any more.

[0137] For each key feature point, the updated three-dimensional vertex coordinates of each key feature point determined in this round of iteration are back-propagated to the three-dimensional face model to transform the coordinates of each key feature point, and then the fourth error of each key feature point is repeatedly determined, and the three-dimensional vertex coordinates of each key feature point are updated with the fourth error until the fourth error no longer decreases, that is, the optimization can no longer be continued, and the optimal result is achieved. In this case, based on the distance error between each key feature point corresponding to the face image and the three-dimensional face model, each key feature point of the three-dimensional face model is constrained, so that the three-dimensional face model after the coordinate transformation is closer to the real face in details, and a high-precision, delicate, and detailed three-dimensional face model is obtained.

[0138] Further, in some examples, see Figure 4 , S207 also includes:

[0139] S2072. Based on each point of the facial point cloud model, iteratively update the three-dimensional vertex coordinates of each point of the three-dimensional face model, and obtain the three-dimensional vertex coordinates of each point after iterative update based on a preset fourth iteration condition.

[0140] Among them, any round of at least one round of iterative update in S2072 includes:

[0141] S2072A. Transform the three-dimensional face model according to the three-dimensional vertex coordinates corresponding to each point of the three-dimensional face model of the current round to obtain a transformed three-dimensional face model.

[0142] In the first iteration, the current three-dimensional vertex coordinates are the three-dimensional vertex coordinates of the three-dimensional face model optimized based on S206 or S2071; in the iterations after the first, the current three-dimensional vertex coordinates are the three-dimensional vertex coordinates updated in the previous iteration.

[0143] S2072B. For any point among the points, determine the fifth error based on the relative distance between the point of the transformed three-dimensional face model and the point of the face point cloud model corresponding to the point.

[0144] In some examples, the point of the transformed three-dimensional face model is the following target point, and the point of the face point cloud model corresponding to the point is the following target nearest point. Based on this, S2072B includes:

[0145] Sub-step 1: Determine each point pair.

[0146] Among them, a point pair includes a target point of a deformed three-dimensional face model and the closest point of the target point in the face point cloud model.

[0147] Sub-step 2: Screen the nearest point of each point pair in each point pair respectively, and determine the target nearest point corresponding to the target point according to the screening result.

[0148] Sub-step three: for any point pair, determine the fifth error based on the relative distance between the target point and the nearest point of the target.

[0149] In sub-step 1 to sub-step 3 of S2072B, since the correspondence between the points of the point cloud and the pixel points of the face image is determined in S203-S204 above, the points of the face point cloud model have semantic information. According to the coordinates of the semantic information of the points of the face point cloud model and the semantic information and coordinates of the three-dimensional face model, the two are aligned, and for any point (target point) of the three-dimensional face model, the N points on the face point cloud model whose relative distance to the projection position of the target point on the face point cloud model is closest to the point are determined as the nearest point of the target point, and then the target point and the nearest point of the target point are determined as a point pair, where N is an integer greater than or equal to 1.

[0150] For a point pair, among the multiple nearest points of the target point, determine a nearest point that meets the screening conditions (for example: the relative distance to the target point is the smallest, or the relative distance to the center point of the area defined by the nearest point is the smallest) as the target nearest point, and determine the fifth error based on the relative distance between the target point and the target nearest point.

[0151] S2072C. Update the three-dimensional vertex coordinates corresponding to the point based on the fifth error.

[0152] For any point of the three-dimensional face model, the fifth error is determined based on the relative distance between the point and the corresponding point on the face point cloud model, and the three-dimensional vertex coordinates of each point of the three-dimensional face model are optimized based on the fifth error, so that each point (target point) of the three-dimensional face model is as close as possible to the corresponding point on the face point cloud model (target nearest point).

[0153] The preset fourth iteration condition is that the value of the fifth error does not decrease any more.

[0154] For each point of the 3D face model, the updated 3D vertex coordinates of each point determined in this round of iterations are back-propagated to the 3D face model to transform the coordinates of each point, and then the fifth error of each point is repeatedly determined, and the 3D vertex coordinates of each point are updated with the fifth error until the fifth error no longer decreases, that is, the optimization can no longer be continued, and the optimal result is achieved. In this case, based on the distance error between the face point cloud model and each corresponding point on the 3D face model, each point of the 3D face model is constrained, so that the 3D face model after the coordinate transformation is closer to the real face in details, and a high-precision, delicate, and detailed 3D face model is obtained.

[0155] Further, in some examples, see Figure 4 , S207 also includes:

[0156] The model point transformation parameters include: pixel point transformation parameters.

[0157] S2073 iteratively updates pixel transformation parameters of each pixel of the rendering of the three-dimensional face model based on each pixel of the rendering of the face image, and obtains pixel transformation parameters of each pixel after iterative update based on a preset fifth iteration condition;

[0158] Among them, any round of at least one round of iterative update in S2073 includes:

[0159] S2073A. Perform pixel transformation processing on each pixel of the rendering image of the three-dimensional face model according to the pixel transformation parameters of the current round of each pixel of the rendering image of the three-dimensional face model to obtain the rendering image of the three-dimensional face model after pixel transformation.

[0160] In the first iteration, the current three-dimensional vertex coordinates are the pixel transformation parameters of each pixel of the three-dimensional face model optimized based on S206 or S2071 or S2075; in the iterations after the first, the current pixel transformation parameters are the pixel transformation parameters updated in the previous round of iteration.

[0161] S2073B. For any pixel point among the pixels, determine a sixth error based on the pixel information of the pixel point in the rendering image of the three-dimensional face model after pixel transformation and the pixel information of the pixel point in the face image corresponding to the pixel point.

[0162] In some examples, S2073B includes:

[0163] Sub-step 1: determining each pixel pair, wherein a pixel pair includes a pixel of a rendering of a three-dimensional face model after pixel transformation and a pixel of a face image corresponding to the pixel of the rendering of the three-dimensional face model after pixel transformation;

[0164] Sub-step 2: for any pair of pixels, determine a sixth error for the pixel based on the difference between the pixel information of the pixel of the face image and the pixel information of the transformed rendering of the three-dimensional face model.

[0165] In sub-step 1 and sub-step 2 of S2073B, the pixel points of the rendering image of the three-dimensional face model and the pixel points of the corresponding face image are aligned through at least one of rotation, translation, scaling, etc., and the pixel points with the same semantics and / or the same position between the two are determined as a pixel pair, and then the difference in pixel information of two key feature points in each pixel pair is calculated, and the sixth error is determined by the difference in pixel information, wherein the essence of the sixth error may be: pixel color error, pixel brightness error, etc.

[0166] S2073C, updating pixel transformation parameters of pixels in the rendering image of the three-dimensional face model after pixel transformation based on the sixth error.

[0167] For any pixel point of the rendering image of the three-dimensional face model, a sixth error is determined based on the difference in pixel information between the pixel point and the pixel point on the corresponding face image, and pixel transformation parameters of each pixel point of the rendering image of the three-dimensional face model are optimized based on the sixth error, so that the pixel points of the rendering image of the three-dimensional face model are as close as possible to the pixel points on the corresponding face image; the essence of the pixel transformation parameters can be: color value, brightness value.

[0168] The preset fifth iteration condition is that the value of the sixth error does not decrease any more.

[0169] For each pixel of the rendering of the three-dimensional face model, the pixel transformation parameters of each pixel after the update determined in this round of iteration are back-propagated to the three-dimensional face model to perform pixel transformation of each pixel, and then the sixth error of each point is repeatedly determined, and the pixel transformation parameters of each point are updated with the sixth error until the sixth error no longer decreases, that is, the optimization can no longer be continued, and the optimal result is achieved. In this case, based on the difference in pixel information of each pixel corresponding to the face image and the rendering of the three-dimensional face model, each pixel of the rendering of the three-dimensional face model is constrained, so that the rendering of the three-dimensional face model after pixel transformation is closer to the texture of the real face in details, and a high-precision, delicate, and real texture detail three-dimensional face model is obtained.

[0170] S208 . Transform each point of the three-dimensional face model according to the model point transformation parameters corresponding to each point after iterative update to obtain a final three-dimensional face model.

[0171] Among them, S205-S208 is an implementation method of S102. Specifically, S102 also has other implementation methods, such as using the SfM method, the SFS method, etc., which are not limited here.

[0172] In the third embodiment disclosed, based on Figure 1 The same principle, Figure 5A face reconstruction device 50 provided in the third embodiment of the present disclosure is shown, and the device comprises:

[0173] A face point cloud model generation module 501 is used to generate a face point cloud model based on face images of a target face from multiple perspectives;

[0174] The 3D face model generation module 502 is used to determine the model transformation parameters of the initial 3D face model based on the face image and the face point cloud model, and transform the initial 3D face model based on the model transformation parameters to obtain a 3D face model corresponding to the target face.

[0175] In some examples, the model transformation parameters include: model overall transformation parameters; the 3D face model generation module 502 includes:

[0176] A first updating submodule is used to iteratively update the overall model transformation parameters of the initial three-dimensional face model based on the face image and the face point cloud model;

[0177] The first transformation submodule is used to transform the initial three-dimensional face model according to the iteratively updated overall model transformation parameters to obtain a three-dimensional face model.

[0178] In some examples, the overall model transformation parameters include: deformation parameters;

[0179] The first update submodule is specifically used for:

[0180] Iteratively updating the deformation parameters of the initial three-dimensional face model for at least one round based on the face image and the face point cloud model, and obtaining the iteratively updated deformation parameters based on a preset first iteration condition;

[0181] Wherein, at least one round of iterative updates includes:

[0182] Deforming the initial three-dimensional face model according to the deformation parameters of the current round to obtain a deformed three-dimensional face model;

[0183] Determine a first error based on key feature points of the face image and key feature points of the deformed three-dimensional face model;

[0184] Determine a second error based on each point of the face point cloud model and each point of the deformed three-dimensional face model;

[0185] updating the deformation parameter based on the first error and the second error;

[0186] The preset first iteration condition is that at least one of the value of the first error and the value of the second error no longer decreases.

[0187] In some examples, the overall model transformation parameters also include: pixel region transformation parameters;

[0188] The first update submodule is specifically used for:

[0189] Iteratively updating the pixel region transformation parameters of the initial three-dimensional face model for at least one round based on the face image and the face point cloud model, and obtaining the iteratively updated pixel region transformation parameters based on a preset second iteration condition;

[0190] Wherein, at least one round of iterative updates includes:

[0191] Performing pixel transformation processing on the rendering of the initial three-dimensional face model according to the pixel region transformation parameters of the current round to obtain a rendering of the three-dimensional face model after pixel transformation;

[0192] Determine a third error based on pixel information of each pixel point of the face image and pixel information of each pixel point of the rendering of the three-dimensional face model after pixel transformation;

[0193] updating a pixel region transformation parameter based on the third error;

[0194] The preset second iteration condition is that the value of the third error does not decrease any more.

[0195] In some examples, the first updating submodule is specifically used to:

[0196] Determine each key feature point pair, wherein a key feature point pair includes a key feature point of a deformed three-dimensional face model and a key feature point of a face image corresponding thereto;

[0197] A first error is determined based on a relative distance between two key feature points of each key feature point pair in each key feature point pair.

[0198] In some examples, the first updating submodule is specifically used to:

[0199] Determine each point pair, wherein a point pair includes a target point of the deformed three-dimensional face model and a closest point of the target point in the face point cloud model;

[0200] Filter the nearest points of each point pair respectively, and update the point pairs according to the filtering results;

[0201] A second error is determined based on the relative distance between the target point and the nearest point of each point pair in the updated point pairs.

[0202] In some examples, the first updating submodule is specifically used to:

[0203] Determine each face region group, wherein a face region group includes a first face region of a rendering of a three-dimensional face model after pixel transformation and a second face region of a face image corresponding to the first face region;

[0204] Determine first area pixels of the first human face area and second area pixels of the second human face area of ​​each human face area pair in each human face area group;

[0205] A third error is determined based on a difference between first region pixels and second region pixels of each face region in each face region group.

[0206] In some examples, the model transformation parameters include: model point transformation parameters; the 3D face model generation module 502 further includes:

[0207] The second updating submodule is used to iteratively update the model point transformation parameters of each point of the three-dimensional face model based on the face image and the face point cloud model;

[0208] The second transformation submodule is used to transform each point of the three-dimensional face model according to the model point transformation parameters corresponding to each point after iterative update to obtain the final three-dimensional face model.

[0209] In some examples, the model point transformation parameters include: three-dimensional vertex coordinates;

[0210] The second update submodule is specifically used for:

[0211] Based on each key feature point of the face image, iteratively update the three-dimensional vertex coordinates of each key feature point of the three-dimensional face model, and obtain the three-dimensional vertex coordinates of each key feature point after iterative update based on a preset third iteration condition;

[0212] Wherein, at least one round of iterative updates includes:

[0213] The three-dimensional face model is transformed according to the three-dimensional vertex coordinates corresponding to the key feature points of each three-dimensional face model in the current round to obtain a transformed three-dimensional face model;

[0214] For any key feature point among the key feature points, determine a fourth error based on a relative distance between the key feature point of the face image and the key feature point of the transformed three-dimensional face model;

[0215] Update the three-dimensional vertex coordinates corresponding to the key feature point based on the fourth error;

[0216] The third iteration condition is that the value of the fourth error does not decrease any more.

[0217] In some examples, the second updating submodule is specifically used to:

[0218] Iteratively updating the three-dimensional vertex coordinates of each point of the three-dimensional face model based on each point of the face point cloud model, and obtaining the three-dimensional vertex coordinates of each point after iterative update based on a preset fourth iteration condition;

[0219] Wherein, at least one round of iterative updates includes:

[0220] The three-dimensional face model is transformed according to the three-dimensional vertex coordinates corresponding to each current point to obtain a transformed three-dimensional face model;

[0221] For any point among the points, determine a fifth error based on a relative distance between the point of the transformed three-dimensional face model and a point of the face point cloud model corresponding to the point;

[0222] The three-dimensional vertex coordinates corresponding to the updating point are updated based on the fifth error;

[0223] The preset fourth iteration condition is that the value of the fifth error does not decrease any more.

[0224] In some examples, the model point transformation parameters include: pixel point transformation parameters;

[0225] The second update submodule is specifically used for:

[0226] Iteratively updating pixel transformation parameters of each pixel of the rendering of the three-dimensional face model based on each pixel of the rendering of the face image, and obtaining pixel transformation parameters of each pixel after iterative update based on a preset fifth iteration condition;

[0227] Wherein, at least one round of iterative updates includes:

[0228] Performing pixel transformation processing on each pixel of the rendering of the three-dimensional face model according to the current pixel transformation parameters of each pixel of the rendering of the three-dimensional face model to obtain the rendering of the three-dimensional face model after pixel transformation;

[0229] For any pixel among the pixels, determining a sixth error based on pixel information of the pixel in the rendering of the three-dimensional face model after pixel transformation and pixel information of the pixel in the face image corresponding to the pixel;

[0230] updating pixel transformation parameters of pixels in the rendering image of the three-dimensional face model after pixel transformation based on the sixth error;

[0231] The preset fifth iteration condition is that the value of the sixth error does not decrease any more.

[0232] In some examples, the 3D face model generation module 502 includes:

[0233] Get submodule, used to get camera parameters;

[0234] The generation submodule is used to generate a face point cloud model based on camera parameters and face images from multiple perspectives.

[0235] In some examples, the camera parameters include camera intrinsic parameters and camera extrinsic parameters, and the 3D face model generation module 502 further includes:

[0236] A projection submodule is used to project the face point cloud model onto face images of multiple perspectives based on camera external parameters, and to project the face point cloud model at each perspective onto the face image corresponding to each perspective based on camera internal parameters;

[0237] The correspondence submodule is used to determine the correspondence between each point of the face point cloud model and each pixel point on the face image.

[0238] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0239] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0240] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0241] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0242] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0243] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as the method of face reconstruction. For example, in some embodiments, the method of face reconstruction may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the method of face reconstruction described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured as a method of face reconstruction in any other appropriate manner (e.g., by means of firmware).

[0244] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0245] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0246] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0247] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0248] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0249] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0250] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0251] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for face reconstruction, the method comprising: Generate a face point cloud model based on face images of the target face from multiple perspectives; Determine the model transformation parameters of the initial three-dimensional face model based on the face image and the face point cloud model, and transform the initial three-dimensional face model based on the model transformation parameters to obtain a three-dimensional face model corresponding to the target face; wherein, when the model transformation parameters include the overall model transformation parameters, specifically include: Iteratively updating overall model transformation parameters of the initial three-dimensional face model based on the face image and the face point cloud model; Transforming the initial three-dimensional face model according to the iteratively updated overall model transformation parameters to obtain the three-dimensional face model; Wherein, the overall transformation parameters of the model include: deformation parameters; The iterative updating of the overall model transformation parameters of the initial three-dimensional face model based on the face image and the face point cloud model includes: Performing at least one round of iterative updating on the deformation parameters of the initial three-dimensional face model based on the face image and the face point cloud model, and obtaining the iteratively updated deformation parameters based on a preset first iterative condition; Wherein, any round of the at least one round of iterative update includes: Performing deformation processing on the initial three-dimensional face model according to the deformation parameters of the current round to obtain a deformed three-dimensional face model; Determine a first error based on key feature points of the facial image and key feature points of the deformed three-dimensional facial model; Determining a second error based on each point of the face point cloud model and each point of the deformed three-dimensional face model; updating the deformation parameter based on the first error and the second error; The initial three-dimensional face model is a three-dimensional deformable face model; and the three-dimensional face model is a face model obtained by optimizing the three-dimensional deformable face model.

2. The method according to claim 1, wherein: The preset first iteration condition is that at least one of the value of the first error and the value of the second error no longer decreases.

3. The method according to claim 1, wherein: The overall model transformation parameters also include: pixel region transformation parameters; The iterative updating of the overall model transformation parameters of the initial three-dimensional face model based on the face image and the face point cloud model also includes: Iteratively updating the pixel region transformation parameters of the initial three-dimensional face model for at least one round based on the face image and the face point cloud model, and obtaining the iteratively updated pixel region transformation parameters based on a preset second iteration condition; Wherein, any round of the at least one round of iterative update includes: Performing pixel transformation processing on the rendering of the initial three-dimensional face model according to the pixel region transformation parameters of the current round to obtain a rendering of the three-dimensional face model after pixel transformation; Determine a third error based on pixel information of each pixel point of the face image and pixel information of each pixel point of the rendering of the three-dimensional face model after the pixel transformation; updating the pixel region transformation parameter based on the third error; The preset second iteration condition is that the value of the third error no longer decreases.

4. The method according to claim 2 or 3, wherein: The determining of the first error based on the key feature points of the face image and the key feature points of the deformed three-dimensional face model includes: Determine each key feature point pair, wherein one of the key feature point pairs includes a key feature point of the deformed three-dimensional face model and a key feature point of the face image corresponding thereto; The first error is determined based on a relative distance between two key feature points of each key feature point pair in each of the key feature point pairs.

5. The method according to any one of claims 2 to 4, wherein: The determining of the second error based on each point of the face point cloud model and each point of the deformed three-dimensional face model includes: Determine each point pair, wherein one of the point pairs includes a target point of the deformed three-dimensional face model and a closest point of the target point in the face point cloud model; Screening the nearest point of each point pair in each point pair respectively, and updating the point pair according to the screening result; The second error is determined based on the relative distance between the target point and the nearest point in each of the updated point pairs.

6. The method according to claim 3, wherein: The determining of the third error based on the pixel information of each pixel point of the face image and the pixel information of each pixel point of the rendering image of the three-dimensional face model after the pixel transformation comprises: Determine each face region group, wherein one of the face region groups includes a first face region of a rendering of the three-dimensional face model after pixel transformation and a second face region of the face image corresponding to the first face region; Determine first area pixels of the first face area and second area pixels of the second face area of ​​each face area pair in each face area group; The third error is determined based on a difference between first region pixels and second region pixels of each face region in each face region group.

7. The method according to any one of claims 1 to 6, wherein: Model transformation parameters include: model point transformation parameters; After the initial 3D face model is transformed according to the iteratively updated overall model transformation parameters to obtain the 3D face model, the model transformation parameters of the initial 3D face model are determined based on the face image and the face point cloud model, and the initial 3D face model is transformed based on the model transformation parameters to obtain a 3D face model corresponding to the target face, further comprising: Iteratively updating model point transformation parameters of each point of the three-dimensional face model based on the face image and the face point cloud model; Each point of the three-dimensional face model is transformed according to the iteratively updated model point transformation parameters of each point to obtain the final three-dimensional face model.

8. The method according to claim 7, wherein: The model point transformation parameters include: three-dimensional vertex coordinates; The iterative updating of the model point transformation parameters of each point of the three-dimensional face model based on the face image and the face point cloud model includes: Iteratively updating the three-dimensional vertex coordinates of each key feature point of the three-dimensional face model based on each key feature point of the face image, and obtaining the three-dimensional vertex coordinates of each key feature point after iterative update based on a preset third iteration condition; Wherein, any round of the at least one round of iterative update includes: Transforming the three-dimensional face model according to the three-dimensional vertex coordinates corresponding to the key feature points of each of the three-dimensional face models in the current round to obtain a transformed three-dimensional face model; For any key feature point among the key feature points, determining a fourth error based on a relative distance between the key feature point of the face image and the key feature point of the transformed three-dimensional face model; Update the three-dimensional vertex coordinates corresponding to the key feature point based on the fourth error; The third iteration condition is that the value of the fourth error does not decrease any more.

9. The method according to claim 8, wherein: The iterative updating of the model point transformation parameters of each point of the three-dimensional face model based on the face image and the face point cloud model also includes: Iteratively updating the three-dimensional vertex coordinates of each point of the three-dimensional face model based on each point of the face point cloud model, and obtaining the iteratively updated three-dimensional vertex coordinates of each point based on a preset fourth iteration condition; Wherein, any round of the at least one round of iterative update includes: Transforming the three-dimensional face model according to the three-dimensional vertex coordinates corresponding to each point of the three-dimensional face model of the current round to obtain a transformed three-dimensional face model; For any point among the points, determining a fifth error based on a relative distance between the point of the transformed three-dimensional face model and a point of the face point cloud model corresponding to the point; Update the three-dimensional vertex coordinates corresponding to the point based on the fifth error; The preset fourth iteration condition is that the value of the fifth error no longer decreases.

10. The method according to claim 9, wherein: The model point transformation parameters also include: pixel point transformation parameters; The iterative updating of the model point transformation parameters of each point of the three-dimensional face model based on the face image and the face point cloud model also includes: Iteratively updating pixel transformation parameters of each pixel of the rendering of the three-dimensional face model based on each pixel of the rendering of the face image, and obtaining the iteratively updated pixel transformation parameters of each pixel based on a preset fifth iteration condition; Wherein, any round of the at least one round of iterative update includes: Performing pixel transformation processing on each pixel of the rendering image of the three-dimensional face model according to the pixel transformation parameters of the current round of each pixel of the rendering image of the three-dimensional face model to obtain the rendering image of the three-dimensional face model after pixel transformation; For any pixel point among the pixels, determining a sixth error based on pixel information of the pixel point in the rendering of the three-dimensional face model after the pixel transformation and pixel information of the pixel point in the face image corresponding to the pixel point; Update the pixel transformation parameter of the pixel in the rendering image of the three-dimensional face model after the pixel transformation based on the sixth error; The preset fifth iteration condition is that the value of the sixth error no longer decreases.

11. The method according to any one of claims 1 to 10, wherein: The step of generating a face point cloud model based on face images of a target face from multiple perspectives includes: Get camera parameters; A facial point cloud model is generated according to the camera parameters and the facial images from the multiple perspectives.

12. The method according to claim 11, wherein: The camera parameters include camera intrinsic parameters and camera extrinsic parameters; After generating the face point cloud model according to the camera parameters and the face images of the multiple perspectives, generating the face point cloud model according to the face images of the target face of the multiple perspectives also includes: Based on the camera external parameters, projecting the face point cloud model onto the face images of the multiple perspectives, and based on the camera internal parameters, projecting the face point cloud model at each perspective onto the face image corresponding to each perspective; Determine the correspondence between each point of the face point cloud model and each pixel point on the face image.

13. A device for face reconstruction, comprising: A face point cloud model generation module is used to generate a face point cloud model based on face images of a target face from multiple perspectives; a 3D face model generation module, configured to determine model transformation parameters of an initial 3D face model based on the face image and the face point cloud model, and transform the initial 3D face model based on the model transformation parameters to obtain a 3D face model corresponding to the target face; When the model transformation parameters include the overall model transformation parameters, the three-dimensional face model generation module includes: A first updating submodule, configured to iteratively update overall model transformation parameters of the initial three-dimensional face model based on the face image and the face point cloud model; A first transformation submodule, configured to transform the initial three-dimensional face model according to the iteratively updated overall model transformation parameters to obtain the three-dimensional face model; The overall transformation parameters of the model include deformation parameters; the first updating submodule is specifically used for: Performing at least one round of iterative updating on the deformation parameters of the initial three-dimensional face model based on the face image and the face point cloud model, and obtaining the iteratively updated deformation parameters based on a preset first iterative condition; Wherein, any round of the at least one round of iterative update includes: Performing deformation processing on the initial three-dimensional face model according to the deformation parameters of the current round to obtain a deformed three-dimensional face model; Determine a first error based on key feature points of the facial image and key feature points of the deformed three-dimensional facial model; Determining a second error based on each point of the face point cloud model and each point of the deformed three-dimensional face model; updating the deformation parameter based on the first error and the second error; The initial three-dimensional face model is a three-dimensional deformable face model; and the three-dimensional face model is a face model obtained by optimizing the three-dimensional deformable face model.

14. The device according to claim 13, wherein: Model transformation parameters include: model point transformation parameters; The three-dimensional face model generation module also includes: A second updating submodule, configured to iteratively update the model point transformation parameters of each point of the three-dimensional face model based on the face image and the face point cloud model; The second transformation submodule is used to transform each point of the three-dimensional face model according to the model point transformation parameters corresponding to each point after iterative update to obtain the final three-dimensional face model.

15. The device according to any one of claims 13-14, wherein: The face point cloud model generation module is specifically used for: Get camera parameters; A facial point cloud model is generated according to the camera parameters and the facial images from the multiple perspectives.

16. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 12.

17. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-12.

18. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 12.

Images