Three-dimensional face reconstruction method and device, computer device and storage medium

Abstract

Embodiments of the present application provide a three-dimensional face reconstruction method and device, computer equipment and a storage medium, relating to the technical field of computer vision. A plurality of target four-channel images corresponding to a target face are obtained, the target four-channel images having different perspectives; the plurality of target four-channel images are input into a pre-trained convolutional neural network to obtain an initial three-dimensional face model, expression coefficients and pose coefficients of the target face, the initial three-dimensional face model being obtained by fusing a three-dimensional average face model and the expression coefficients; texture coefficients of the target face are determined according to the plurality of target four-channel images; and the pose coefficients and the texture coefficients are fused with the initial three-dimensional face model to obtain a three-dimensional face corresponding to the target face. The plurality of depth images with different perspectives can be comprehensively utilized to achieve high-precision reconstruction of a three-dimensional face model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a three-dimensional face reconstruction method and device, computer equipment and storage medium. BACKGROUND

[0002] With the development of computer and artificial intelligence technology, three-dimensional (3D) reconstruction technology has shown broad application prospects. Three-dimensional reconstruction refers to the process of establishing a mathematical model suitable for computer representation and processing of 3D objects. Among them, three-dimensional face reconstruction is widely used in security, three-dimensional animation and other fields, making it a research direction with practical significance.

[0003] At present, in three-dimensional face reconstruction, a three-dimensional face model is mainly reconstructed from a single image. Specifically, it can be divided into two kinds: a three-dimensional face reconstruction method based on single-view RGB image and a three-dimensional face reconstruction method based on single-view RGBD image. Among them, the three-dimensional face reconstruction method based on single-view RGB image takes a single image containing only R, G and B channels as an input sample to reconstruct a three-dimensional face model, which has good modeling effect under the condition that detailed face information of the object is difficult to obtain. The three-dimensional face reconstruction method based on single-view RGBD image can obtain an RGBD image by recording the distance from the sensor to the measured object in each pixel value in a manner similar to a grayscale image, and then take the RGBD image as an input sample. In this way, a three-dimensional face model can be reconstructed from a small amount of storage. Figure Three Figure Four However, the above two methods of reconstructing a three-dimensional face model from a single image, whether containing depth information or not, lack semantic information for the occluded part of the reconstructed three-dimensional model, and can only obtain data for the occluded part by estimation, thereby affecting the reconstruction accuracy.

[0004] The purpose of the present application includes, for example, providing a three-dimensional face reconstruction method, device, computer equipment and storage medium, which can comprehensively utilize multiple depth images of different perspectives to realize high-precision reconstruction of a three-dimensional face model. SUMMARY

[0005] The purpose of the present application includes, for example, providing a three-dimensional face reconstruction method, device, computer equipment and storage medium, which can comprehensively utilize multiple depth images of different perspectives to realize high-precision reconstruction of a three-dimensional face model.

[0006] Embodiments of the present application can be implemented as follows:

[0007] In a first aspect, the present application provides a three-dimensional face reconstruction method, which comprises:

[0008] ​obtaining a plurality of target four-channel images corresponding to a target face, the target four-channel images being different in perspective;

[0009] inputting the plurality of target four-channel images into a pre-trained convolutional neural network to obtain an initial three-dimensional face model and expression coefficients and pose coefficients of the target face, the initial three-dimensional face model being obtained by fusing a three-dimensional average face model and the expression coefficients;

[0010] determining texture coefficients of the target face according to the plurality of target four-channel images;

[0011] fusing the pose coefficients and the texture coefficients with the initial three-dimensional face model to obtain a three-dimensional face corresponding to the target face.

[0012] In an optional implementation, the inputting the plurality of target four-channel images into a pre-trained convolutional neural network to obtain an initial three-dimensional face model and expression coefficients and pose coefficients of the target face includes:

[0013] inputting the plurality of target four-channel images into the convolutional neural network to obtain intermediate layer features of each of the target four-channel images;

[0014] splicing the intermediate layer features to obtain spliced intermediate layer features;

[0015] inputting the spliced intermediate layer features into a model subnetwork of the convolutional neural network to obtain the initial three-dimensional face model and the expression coefficients;

[0016] inputting each of the intermediate layer features into a pose regression subnetwork of the convolutional neural network to obtain the pose coefficients.

[0017] In an optional implementation, the determining texture coefficients of the target face according to the plurality of target four-channel images includes:

[0018] establishing a Gaussian pyramid model, a Laplacian pyramid model and a mask pyramid model according to each of the target four-channel images, respectively;

[0019] determining the texture coefficients according to the Gaussian pyramid model, the Laplacian pyramid model and the mask pyramid model.

[0020] In an optional implementation, the determining the texture coefficients according to the Gaussian pyramid model, the Laplacian pyramid model and the mask pyramid model includes:

[0021] generating a hybrid Gaussian pyramid according to the Gaussian pyramid and the mask pyramid;

[0022] generating a mixed Laplacian pyramid according to the Laplacian pyramid and the mask pyramid;

[0023] adding the mixed Gaussian pyramid and the mixed Laplacian pyramid, and upsampling to obtain a sampling pyramid;

[0024] adding the sampling pyramid and the Laplacian pyramid to determine a texture coefficient.

[0025] In an optional implementation, the acquiring the plurality of target four-channel images corresponding to the target face comprises:

[0026] acquiring a video sequence corresponding to the target face, the video sequence comprising the plurality of candidate four-channel images corresponding to the target face;

[0027] determining pose data of each candidate four-channel image, the pose data comprising a pitch angle, a yaw angle and a roll angle;

[0028] selecting the plurality of target four-channel images from the plurality of candidate four-channel images according to the pose data of each candidate four-channel image and preset pose combination information.

[0029] In an optional implementation, the determining the pose data of each candidate four-channel image comprises:

[0030] performing key point detection on a first candidate four-channel image to obtain a first point set, wherein the first candidate four-channel image is any candidate four-channel image in the plurality of candidate four-channel images;

[0031] taking a set of key points corresponding to a three-dimensional average face model as a second point set, and determining a correspondence between each first point set and the second point set;

[0032] pairing the first point set and the second point set to obtain a plurality of point set pairs, each point set pair comprising one key point in the first point set and one key point in the second point set;

[0033] determining dimension transformation information corresponding to each point set pair respectively;

[0034] determining the pose data of the first candidate four-channel image according to the dimension transformation information corresponding to each point set pair.

[0035] In an optional implementation, the method further comprises:

[0036] The initial convolutional data network is trained based on the target loss function to obtain the convolutional neural network. The target loss function includes at least one of the following: key point to epipolar distance loss function, depth data loss function, point cloud nearest neighbor distance loss function, image consistency loss function, and face point loss function.

[0037] Secondly, this application provides a three-dimensional face reconstruction device, the device comprising:

[0038] The acquisition module is used to acquire multiple target four-channel images corresponding to the target face, wherein the viewpoints of the target four-channel images are different;

[0039] The processing module is used to input the multiple target four-channel images into a pre-trained convolutional neural network to obtain an initial three-dimensional face model and the expression coefficients and pose coefficients of the target face. The initial three-dimensional face model is obtained by fusing the three-dimensional average face model with the expression coefficients.

[0040] The determination module is used to determine the texture coefficients of the target face based on the multiple target four-channel images;

[0041] The fusion module is used to fuse the pose coefficients and texture coefficients with the initial 3D face model to obtain the 3D face corresponding to the target face.

[0042] The processing module is further configured to: input the multiple target four-channel images into the convolutional neural network to obtain intermediate layer features of each target four-channel image; concatenate the intermediate layer features to obtain concatenated intermediate layer features; input the concatenated intermediate layer features into the model subnetwork of the convolutional neural network to obtain the initial three-dimensional face model and the expression coefficients; and input the intermediate layer features into the pose regression subnetwork of the convolutional neural network to obtain the pose coefficients.

[0043] The determining module is further configured to: establish a Gaussian pyramid model, a Laplacian pyramid model, and a mask pyramid model based on each of the target four-channel images; and determine the texture coefficients based on the Gaussian pyramid model, the Laplacian pyramid model, and the mask pyramid model.

[0044] The determining module is further configured to: generate a mixed Gaussian pyramid based on the Gaussian pyramid and the mask pyramid; generate a mixed Laplacian pyramid based on the Laplacian pyramid and the mask pyramid; add the mixed Gaussian pyramid and the mixed Laplacian pyramid and upsample them to obtain a sampling pyramid; and add the sampling pyramid and the Laplacian pyramid to determine the texture coefficients.

[0045] The acquisition module is specifically further configured to acquire a video sequence corresponding to the target face, the video sequence including a plurality of to-be-selected four-channel images corresponding to the target face; determine pose data of each to-be-selected four-channel image, the pose data including a pitch angle, a yaw angle, and a roll angle; and select the plurality of target four-channel images from the plurality of to-be-selected four-channel images according to the pose data of each to-be-selected four-channel image and preset pose combination information.

[0046] The acquisition module is specifically further configured to perform key point detection on a first to-be-selected four-channel image to obtain a first point set, wherein the first to-be-selected four-channel image is any to-be-selected four-channel image in the plurality of to-be-selected four-channel images; take a set of key points corresponding to a three-dimensional average face model as a second point set, and determine a correspondence between each first point set and the second point set; pair the first point set and the second point set to obtain a plurality of point set pairs, each point set pair including one key point in the first point set and one key point in the second point set; determine dimension transformation information corresponding to each point set pair respectively; and determine pose data of the first to-be-selected four-channel image according to the dimension transformation information corresponding to each point set pair.

[0047] The training module is configured to train the initial convolutional neural network based on a target loss function to obtain the convolutional neural network, wherein the target loss function includes at least one of a key point-to-epipolar line distance loss function, a depth data loss function, a point cloud nearest neighbor distance loss function, an image consistency loss function, and a face point loss function.

[0048] In a third aspect, a computer device is provided, which includes a processor, a storage medium, and a bus. The storage medium stores machine readable instructions executable by the processor. When the computer device is running, the processor and the storage medium communicate through the bus. The processor executes the machine readable instructions to perform the steps of the three-dimensional face reconstruction method according to any one of the preceding embodiments.

[0049] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of the three-dimensional face reconstruction method according to any one of the preceding embodiments are implemented.

[0050] The beneficial effects of the embodiments of the present application include:

[0051] Using the 3D face reconstruction method, apparatus, computer equipment, and storage medium provided in this application, firstly, this application reconstructs the 3D face corresponding to the target face by inputting multiple target four-channel images from different perspectives into a pre-trained convolutional neural network. Compared with face reconstruction methods based on a single view, this method enables the reconstructed 3D face to have more feature point information of the target face, thus improving reconstruction accuracy. Secondly, by calculating expression coefficients, pose coefficients, and texture coefficients separately and then fusing them to obtain the 3D face, this method is easy to implement in parallel computation, has high real-time performance, and provides good expressive effects. Attached Figure Description

[0052] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is an overall flowchart of the three-dimensional face reconstruction method provided in the embodiments of this application;

[0054] Figure 2 A flowchart illustrating the steps of the three-dimensional face reconstruction method provided in this application embodiment;

[0055] Figure 3 A schematic diagram illustrating another step of the three-dimensional face reconstruction method provided in this application embodiment;

[0056] Figure 4 A schematic diagram illustrating another step of the three-dimensional face reconstruction method provided in this application embodiment;

[0057] Figure 5 A schematic diagram illustrating another step of the three-dimensional face reconstruction method provided in this application embodiment;

[0058] Figure 6 A schematic diagram illustrating another step of the three-dimensional face reconstruction method provided in this application embodiment;

[0059] Figure 7 A schematic diagram illustrating another step of the three-dimensional face reconstruction method provided in this application embodiment;

[0060] Figure 8 This is a schematic diagram of the structure of the three-dimensional face reconstruction device provided in the embodiments of this application;

[0061] Figure 9 A schematic diagram of the structure of a computer device provided in an embodiment of this application.

[0062] Icon: 101 - multiple four-channel image or video streams to be selected; 102 - three-dimensional average face model; 103 - convolutional neural network; 1031 - expression coefficient; 1032 - pose coefficient; 104 - texture coefficient; 105 - three-dimensional face corresponding to the target face; 106 - target loss function; 10 - three-dimensional face reconstruction device; 1001 - acquisition module; 1002 - processing module; 1003 - determination module; 1004 - fusion module; 1005 - training module; 2001 - processor; 2002 - memory. DETAILED DESCRIPTION

[0063] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the following will be a clear and complete description of the technical solutions in the embodiments of the present application in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various different configurations.

[0064] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. All other embodiments obtained by a person of ordinary skill in the art without making creative efforts based on the embodiments in the present application are within the scope of protection of the present application.

[0065] It should be noted that: similar reference numbers and letters represent similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0066] In addition, if the terms "first", "second", etc. are used only to distinguish description, and cannot be understood as indicating or implying relative importance.

[0067] It should be noted that: the features in the embodiments of the present application can be combined with each other without conflict.

[0068] Three-dimensional face reconstruction based on two-dimensional images is an important research content in the field of computer graphics and computer vision. In the current work, three-dimensional face reconstruction is mainly based on single image, which is mainly divided into three-dimensional face reconstruction method based on RGB image and three-dimensional face reconstruction method based on RGBD image. Although the single-view-based three-dimensional face reconstruction method has achieved good performance in the case of normal face angle, whether it contains depth information or not, due to the reconstruction of only one view image, the semantic information of the occluded part is lacking, and the data of the occluded part can only be obtained by estimation, thereby affecting the reconstruction accuracy.

[0069] Based on this, the applicant proposes a three-dimensional face reconstruction method, device, computer equipment and storage medium, which can input multiple target four-channel images with different perspectives into a pre-trained convolutional neural network to reconstruct the corresponding three-dimensional face. Compared with the single-view three-dimensional face reconstruction method, more three-dimensional face feature point information of multiple angles can be obtained, the problem of occlusion of single-view face images can be avoided, the accuracy of face reconstruction can be improved, and further, through the calculation of expression coefficients, pose coefficients and texture coefficients respectively, the three-dimensional face is finally fused, which is easy to realize parallel computing, has high real-time performance and good expression effect.

[0070] The three-dimensional face reconstruction method, device, computer equipment and storage medium provided by the embodiment of the application are explained and described as follows in combination with a plurality of specific application examples.

[0071] RGB color mode is an industry standard for color, which is obtained by changing and superimposing three color channels of red (R), green (G) and blue (B). The RGBD image includes a common RGB three-channel color image and a four-channel image of a depth map (Depth Map). The depth map is similar to a gray image, except that each pixel value is the actual distance from the sensor to the object. Usually, the RGB image and the depth map are registered, so that there is a one-to-one relationship between the pixel points.

[0072] In the present application, multiple RGBD images or video streams with depth information are taken as input, and a three-dimensional face reconstruction method based on multiple perspective RGB images is provided, and the flowchart of the method is as shown in Figure 1

[0073] First, the obtained multiple four-channel images or video streams 101 are subjected to face key point detection, and the detected face key points corresponding to each RGBD image are taken as a first point set of each RGBD image, and the three-dimensional face key points in the three-dimensional average face model 102 are taken as a second point set. According to the index correspondence relationship between the first point set and the second point set, the first point set and the second point set are paired, and then the loss function of the key points in the paired first point set and the key points in the second point set is iteratively optimized according to the nonlinear least squares method, so as to estimate the pose data of each RGBD image, including the pitch angle, the yaw angle and the roll angle.

[0074] ​Then, three RGBD images closest to preset pose combination information are selected from the above RGBD images as inputs of the convolutional neural network 103 to reconstruct the three-dimensional face. The preset pose combination information includes three sets of different target pose data. The convolutional neural network 103 first obtains the middle layer features of the three RGBD images, inputs the features into a model subnetwork of the convolutional neural network 103 after splicing, and obtains an initial three-dimensional face model and an expression coefficient 1031. The initial three-dimensional face model is obtained by fusing the three-dimensional average face model 102 and the expression coefficient 1031. Then, the middle layer features of the three RGBD images are input into a pose regression subnetwork of the convolutional neural network 103 to obtain a pose coefficient 1032 of the target face. Next, the texture coefficient 104 determined by the three RGBD images is fused with the initial three-dimensional face model, the pose coefficient 1032, and the texture coefficient 104 to obtain a three-dimensional face 105 corresponding to the target face.

[0075] The above process is the process of reconstructing the three-dimensional face after the convolutional network is trained. After the three-dimensional face 105 corresponding to the target face is reconstructed, the convolutional neural network 103 renders the face according to the initial three-dimensional face model, the pose coefficient 1032, and the texture coefficient 104 obtained in the above steps to obtain three corresponding reconstructed images. Based on the target loss function 106, the three reconstructed images and the input three RGBD images are used to optimize the parameters of the convolutional neural network 103 in the gradient backpropagation manner from the key point to the polar line distance loss function, the depth data loss function, the point cloud nearest neighbor distance loss function, the image consistency loss function, and the face point loss function in the target loss function 106. The training process is repeatedly performed until the parameters of the convolutional neural network 103 converge to a stable value, and the training of the convolutional neural network 103 is completed.

[0076] Figure 2 A step flowchart of a three-dimensional face reconstruction method provided by an embodiment of the application is shown. The execution subject of the method can be a computer device with computing processing capability. As shown in the figure, Figure 2 The method includes the following steps.

[0077] S201, obtaining a plurality of target four-channel images corresponding to a target face. The target four-channel images have different viewing angles.

[0078] The target four-channel images can be RGBD images. The plurality of RGBD images include images of different viewing angles of the same target face, which are obtained by screening from inputted plurality of RGBD images or video streams containing depth information.

[0079] In the following steps in the embodiments of the present application, three target four-channel images are taken as an example for illustration.

[0080] S202, input the plurality of target four-channel images into the pre-trained convolutional neural network to obtain an initial three-dimensional face model and expression coefficients and pose coefficients of the target face.

[0081] The initial three-dimensional face model is obtained by fusing the three-dimensional average face model and the expression coefficients.

[0082] The three RGB images corresponding to the three target four-channel images are input into the pre-trained convolutional neural network, the pre-trained convolutional neural network can calculate the expression coefficients of the target face, the identity coefficients of the face, and fuse the expression coefficients with the three-dimensional average face model to obtain the initial three-dimensional face model with expression coefficients. The identity coefficients of the face are used to describe the features of the face, such as face width, forehead height, and other special features, for example, dimples.

[0083] The three-dimensional average face model is a variable face model (3D Morphable Face Model, 3DMM for short), the main idea of which is that if there is a certain correspondence between two images, a smooth transition effect image can be generated using the correspondence. That is, 3DMM represents a three-dimensional average face model generated from a plurality of face images, and the convolutional neural network model can calculate a set of parameter information of the three-dimensional average face model and the target face to describe the deviation between them, so that the three-dimensional average face model can be transformed according to the parameter information to obtain a three-dimensional face corresponding to the target face. The parameter information can include expression coefficients, pose coefficients, and texture coefficients.

[0084] The expression coefficients are used to describe the change of one or more states of the facial muscles.

[0085] In addition, the convolutional neural network can also calculate the pose coefficients of the three-dimensional face according to the three RGB images. Alternatively, if the pose of the three-dimensional average face model is taken as a standard angle, the pose coefficients can be used to describe the orientation deviation of the target face relative to the standard angle.

[0086] S203, determine the texture coefficients of the target face according to the plurality of target four-channel images.

[0087] The texture coefficients of the target face can be extracted by processing the three RGBD images to describe the texture features of the target face.

[0088] The texture feature is a visual feature reflecting the homogeneity in the face image, and embodies the surface structure organization arrangement attribute with slow change or periodic change on the face surface. The texture feature can be expressed by the gray scale distribution of the pixel and the spatial neighborhood thereof.

[0089] S204, fusing the pose coefficient and the texture coefficient with the initial three-dimensional face model to obtain a three-dimensional face corresponding to the target face.

[0090] According to the above description, the pose coefficient and the texture coefficient are both additional parameters used for describing the three-dimensional face corresponding to the target face and the initial three-dimensional face model. According to the pose coefficient and the texture coefficient, the parameters used for describing the pose and the texture in the initial three-dimensional face model are fused to obtain the three-dimensional face corresponding to the target face.

[0091] In the embodiment, the three-dimensional face reconstruction is performed by using multiple target four-channel images, so that the convolutional neural network can learn and use the multi-view face feature information to realize the three-dimensional face reconstruction, and the reconstruction accuracy is improved. In addition, the pose coefficient, the texture coefficient and the expression coefficient can be calculated simultaneously, and the real-time performance is improved.

[0092] It should be noted that in the above embodiment, the multiple target four-channel images corresponding to the target face can be directly inputted by a user as the target four-channel images according to the pre-set pose and quantity rules. Three four-channel images of the target face with the required pose data can be obtained as the target four-channel images. Alternatively, multiple four-channel images of the target face in different poses can be inputted as the candidate four-channel images, and the three four-channel images of the target face with the required pose data can be obtained as the target four-channel images by screening through the following steps S302 to S303. Alternatively, a video stream containing the target face and having depth information can be directly inputted. When the input information is the video stream, the process of obtaining the target four-channel images is described in the following embodiment.

[0093] Optionally, as shown in Figure 3 the above step S201, the multiple target four-channel images corresponding to the target face can be obtained by the following steps S301 to S303.

[0094] S301, obtaining a video sequence corresponding to the target face, and the video sequence includes multiple candidate four-channel images corresponding to the target face.

[0095] The video sequence corresponding to the target face can be split into multiple frames of images as the multiple candidate four-channel images corresponding to the target face, or multiple RGBD images with different views can be directly inputted as the multiple candidate four-channel images corresponding to the target face. The video stream or the images to be inputted satisfy that the information of different angles of the same face is included, and the image depth information is contained. The specific image input form is not limited in the present application.

[0096] S302, determine the pose data of each candidate four-channel image, the pose data including: a yaw angle, a pitch angle and a roll angle.

[0097] The pose data is used to represent the orientation of each target face image in the actual three-dimensional space in each two-dimensional four-channel image. The orientation can be represented by three different direction rotation angles, wherein the yaw angle represents the rotation angle of the target face about the X axis, the pitch angle represents the rotation angle of the target face about the Y axis, and the roll angle represents the rotation angle of the target face about the Z axis.

[0098] That is, the above-mentioned pose data can describe the orientation difference of each target face with respect to the three-dimensional average face model on the X axis, the Y axis and the Z axis respectively. According to the adjustment of the three-dimensional average face model according to the pose data, the three-dimensional average face model can be adjusted to the corresponding pose of each target face. Therefore, the orientation difference can be quantified by an algorithm, so as to determine the pose data of each candidate four-channel image.

[0099] S303, according to the pose data of each candidate four-channel image and the preset pose combination information, screen a plurality of target four-channel images from the plurality of candidate four-channel images.

[0100] The preset pose combination information can include the corresponding pose data of the left side of the face, the right side of the face and the front face of the face respectively, wherein the corresponding pose data of the right side of the face is: yaw=-45, pitch=0, roll=0, the corresponding pose data of the left side of the face is: yaw=45, pitch=0, roll=0, and the corresponding pose data of the front face is: yaw=45, pitch=0, roll=0.

[0101] Next, a group of three target four-channel images closest to the preset pose combination information can be determined by screening the pose data corresponding to each candidate four-channel image obtained in the above steps. It can be understood that the pose data corresponding to the above-mentioned three target four-channel images is the candidate four-channel image with the smallest difference from the corresponding pose data of the left side of the face, the right side of the face and the front face of the face in the pose data of each candidate four-channel image.

[0102] In this embodiment, by screening the pose data of the plurality of candidate four-channel images calculated, a plurality of target four-channel images are determined, so as to provide as comprehensive face information as possible for the three-dimensional face reconstruction in the following steps, and improve the accuracy of the reconstruction.

[0103] Optionally, as Figure 4As shown, in step S302, determining the pose data of each candidate four-channel image can include the following steps.

[0104] S401, performing key point detection on the first candidate four-channel image to obtain a first point set.

[0105] The first candidate four-channel image is any one of the plurality of candidate four-channel images.

[0106] Face key point detection refers to a method of locating the positions of key regions of a face in a given face image, including the locations of eyebrows, eyes, nose, mouth, and face contour. Face key point detection algorithms include global-based methods, graph model-based methods, hierarchical regression-based methods, and deep learning-based methods, etc., which are not limited herein.

[0107] For each candidate four-channel image, a face key point detection algorithm can be used to obtain a plurality of key points for describing the target face on each candidate four-channel image, and the plurality of key points of each candidate four-channel image are taken as a first point set corresponding to the candidate four-channel image. It should be noted that the number of key points can be set to different numbers according to different face key point detection algorithms, which are not limited herein.

[0108] S402, taking the set of key points corresponding to the three-dimensional average face model as a second point set, and determining the correspondence between each first point set and the second point set.

[0109] The first point set is a series of face key points on a two-dimensional plane, and the second point set is a face key point in a three-dimensional space. The number of the first point set and the number of the second point set can be equal, so that the index correspondence between the first point set and the second point set can be established according to the sequence numbers of the key points in the first point set and the sequence numbers of the key points in the second point set.

[0110] S403, pairing the first point set and the second point set to obtain a plurality of point set pairs, each point set pair including one key point in the first point set and one key point in the second point set.

[0111] It can be understood that the key points in the second point set are projected on a two-dimensional plane, and the key points in the first point set and the second point set corresponding to the points on the nearest projection plane establish an index correspondence to form a point set pair.

[0112] S404, respectively determining the dimension transformation information corresponding to each point set pair.

[0113] The dimension transformation information can be understood as how to perform rotation, translation, stretching, etc. on the key points in the first point set so that the key points in the first point set coincide with the key points in the second point set. Quantifying the process of the above transformation operation is the dimension transformation information, specifically, can include the process that the first point set is transformed into the second point set by a rotation matrix, a translation matrix and a scaling scale.

[0114] S405, according to the dimension transformation information corresponding to each point set, determining the pose data of the first candidate four-channel image.

[0115] The nonlinear least squares method can be used to estimate the pose data of each candidate four-channel image through the square difference error loss function of the following formula, specifically, can include: yaw, pitch, roll.

[0116]

[0117] Wherein, n represents the number of key points of the first point set and the second point set on the candidate four-channel image, q i represents the key point of the i th first point set, P i represents the key point of the i th second point set, q i and P i is a point set pair. R is a rotation matrix composed of yaw, pitch and roll, T is a translation matrix, and S is a scaling scale for pose estimation of the first point set.

[0118] The above function solves the minimum transformation angle between the plurality of point set pairs composed of the key points in the first point set and the corresponding key points in the second point set, which can be solved by gradient descent method through multiple iterations, yaw, pitch and roll that can meet the condition, when the solution converges to a stable value, the whole optimization process is completed. In turn, the yaw, pitch and roll corresponding to each candidate four-channel image are solved.

[0119] In this embodiment, the yaw, pitch and roll corresponding to each candidate four-channel image are solved by nonlinear least squares method estimation, and the pose estimation of each candidate four-channel image is completed, thereby providing a selection space for selecting the target four-channel image.

[0120] Optionally, as Figure 5 shown, the above step S202, inputting the plurality of target four-channel images into the convolutional neural network trained in advance to obtain the initial face three-dimensional model and the expression coefficient and the pose coefficient of the target face, can be realized by the following steps S501 to S504.

[0121] S501, input the plurality of target four-channel images into the convolutional neural network to obtain intermediate layer features of the target four-channel images.

[0122] The RGB image in the three target four-channel images obtained above can be input into the convolutional neural network. The first layer of the convolutional neural network first extracts features from the RGB image to obtain intermediate layer feature descriptions for the three target four-channel images respectively.

[0123] The intermediate layer features can be further feature extraction of the input RGB image, and the ultimate goal is to exclude the interference of other features and obtain feature descriptors that are more important for three-dimensional face reconstruction of the convolutional neural network as much as possible.

[0124] S502, splice the intermediate layer features to obtain spliced intermediate layer features.

[0125] The intermediate layer features of the convolutional neural network corresponding to each RGB image are spliced to obtain complete features of the target face, i.e., the spliced intermediate layer features.

[0126] S503, input the spliced intermediate layer features into the model subnetwork of the convolutional neural network to obtain an initial three-dimensional face model and an expression coefficient.

[0127] It can be understood that the model subnetwork is part of the convolutional neural network, and the spliced intermediate layer features can be input into the trained model subnetwork to output the initial three-dimensional face model and the expression coefficient.

[0128] The initial three-dimensional face model is obtained by fusing the three-dimensional average face model and the expression coefficient.

[0129] Optionally, the convolutional neural network can output an identity coefficient of the face in addition to the initial three-dimensional face model and the expression coefficient according to the spliced intermediate layer features.

[0130] S504, input the intermediate layer features into the pose regression subnetwork of the convolutional neural network to obtain a pose coefficient.

[0131] The pose regression subnetwork is also part of the convolutional neural network. The pose regression subnetwork can obtain the pose coefficient of the target face in the three-dimensional space relative to the average face model, including yaw, pitch, and roll, according to the intermediate layer features of multiple views.

[0132] In this embodiment, by splicing the intermediate layer features and then inputting them into the model subnetwork, the face features of multiple views can be learned, and the initial three-dimensional face model obtained is more accurate compared to other methods of splicing in the neural network.

[0133] Optionally, as shown in Figure 6 In step S203, the texture coefficients of the target face are determined according to the plurality of target four-channel images, which can be implemented by steps S601-S602.

[0134] S601, a Gaussian pyramid model, a Laplacian pyramid model and a mask pyramid model are respectively established according to each target four-channel image.

[0135] In the image processing process, the same image generally contains useful information at different scales. The finer the scale, the richer the image detail information. In order to fully and effectively utilize these information, it is usually necessary to perform multi-scale description on the image. The image pyramid is a classic image multi-scale description method. It is an effective and concise structure for describing images in multiple resolutions. The pyramid of an image is actually a series of images arranged in the shape of a pyramid with gradually reduced resolutions and all from the same original picture. Each image in the set is called a layer. The bottom layer of the image pyramid is a high-resolution representation of the source image, and the top layer is a low-resolution approximation. The stacked images like a "pyramid" with higher levels and smaller images have lower resolutions. It is established by a series of iterative calculations. Therefore, the image pyramid can better describe the image information in a multi-scale manner.

[0136] The Gaussian pyramid is obtained by using Gaussian smoothing and downsampling to obtain the image of the upper layer, and then repeatedly iterating several times to form an image data structure with a pyramid shape, i.e., the Gaussian pyramid.

[0137] The Laplacian pyramid is a reconstruction of the Gaussian pyramid. It is the difference between each layer image of the Gaussian pyramid and the predicted image obtained by upsampling and Gaussian convolution of the upper layer image.

[0138] The mask is used to shield the selected image, pattern or object from the image processing (globally or locally) to control the area or process of image processing. The mask pyramid is used to identify the position information of the fusion of the above-mentioned Gaussian pyramid and Laplacian pyramid.

[0139] Taking three target four-channel images as an example, a Gaussian pyramid, a Laplacian pyramid and a mask pyramid can be established for each target four-channel image. Optionally, a Gaussian pyramid can be established for the left side of the target four-channel image with the attitude coefficient yaw=45, pitch=0, roll=0, a Laplacian pyramid can be established for the right side of the target four-channel image with the attitude coefficient yaw=-45, pitch=0, roll=0, and finally a mask pyramid can be established for the front face of the target four-channel image with the attitude coefficient yaw=45, pitch=0, roll=0.

[0140] S602, determine the texture coefficient according to the Gaussian pyramid model, the Laplacian pyramid model and the mask pyramid model.

[0141] Finally, the Gaussian pyramid and the Laplacian pyramid can be fused according to the mask pyramid model to obtain the texture coefficient of the target face.

[0142] In the embodiment, the texture coefficient of the target face is determined by establishing the Gaussian pyramid, the Laplacian pyramid and the mask pyramid for the target four channels respectively, and then fusing them. The texture coefficient obtained in this way has higher accuracy and better expression effect.

[0143] Optionally, as shown in the step S602, the texture coefficient can be determined according to the Gaussian pyramid model, the Laplacian pyramid model and the mask pyramid model, which can be realized by the following steps S701 to S704. Figure 7

[0144] S701, generate a mixed Gaussian pyramid according to the Gaussian pyramid and the mask pyramid.

[0145] The multi-layer images of the Gaussian pyramid are added to the multi-layer images of the mask pyramid respectively to obtain the mixed Gaussian pyramid. Here, the mask pyramid is used to identify the position of the mixed Gaussian pyramid for fusion, and the mixed Gaussian pyramid is used to describe the features of the left half of the target face.

[0146] S702, generate a mixed Laplacian pyramid according to the Laplacian pyramid and the mask pyramid.

[0147] The same as the generation step of the mixed Gaussian pyramid, the multi-layer images of the Laplacian pyramid can be added to the multi-layer images of the mask pyramid respectively to generate the mixed Laplacian pyramid. Here, the mask pyramid is used to identify the position of the mixed Laplacian pyramid for fusion, and the mixed Laplacian pyramid is used to describe the features of the right half of the target face.

[0148] S703, add the mixed Gaussian pyramid and the mixed Laplacian pyramid, and up-sample to obtain a sampling pyramid.

[0149] The addition of the mixed Gaussian pyramid and the mixed Laplacian pyramid is performed on each spatial frequency layer respectively, so that different fusion operators can be used for addition according to the features and details on different layers, and then up-sampling is performed to achieve the purpose of fusing the mixed Gaussian pyramid and the mixed Laplacian pyramid.

[0150] S704, add the sampling pyramid and the Laplacian pyramid to determine the texture coefficient.

[0151] ​The mixed Gaussian pyramid is added to the mixed Laplace pyramid, the sampling pyramid obtained after upsampling is added to the corresponding layer of the Laplace pyramid, and the final fusion result is taken as the texture coefficient of the target face.

[0152] In the embodiment, the texture coefficient is obtained by fusing the mixed Gaussian pyramid, the mixed Laplace pyramid, and the Laplace pyramid after upsampling. Compared with the prior art, the precision of the texture coefficient is higher, and the texture is more similar to that in the target four-channel image of the target face.

[0153] Optionally, the three-dimensional face reconstruction method provided in the embodiment of the application further includes: training the initial convolutional neural network based on a target loss function to obtain the convolutional neural network, wherein the target loss function includes at least one of the following: a key point to epipolar line distance loss function, a depth data loss function, a point cloud nearest neighbor distance loss function, an image consistency loss function, and a face point loss function.

[0154] Optionally, three reconstructed images corresponding to the three selected four-channel images can be obtained by the initial face three-dimensional model, the pose coefficient, and the texture coefficient. Then, the loss value is calculated according to the target loss function, and finally, the parameters of the initial convolutional neural network are updated in the gradient backpropagation manner to obtain the trained convolutional neural network. In the application, the target loss function can include: a key point to epipolar line distance loss function, a depth data loss function, a point cloud nearest neighbor distance loss function, an image consistency loss function, and a face point loss function.

[0155] Since the key points in a pair of images will fall on the epipolar line, the key point to epipolar line distance loss function can be used to constrain the pose learned by the convolutional neural network. The key point to epipolar line distance loss function is used to calculate the distance between the corresponding points in the selected four-channel image and the reconstructed image and the epipolar line. It can be denoted as:

[0156]

[0157] E = t 1→2 × R 1→2 , t 1→2 represents the translation vector from view 1 to view 2, R 1→2 represents the rotation vector from the first view to the second view, N represents the number of different views, that is, the number of selected four-channel images and reconstructed images, K represents the number of key points, p j1 represents the jth key point in the first view, p j2 represents the jth key point in the second view, Ep j(1) represents the first component of the epipolar line, Ep j(2)The second component represents the polar line. Wherein the first view angle can be the view angle of the to-be-selected four-channel image, and the second view angle can be the view angle of the reconstructed image.

[0158] Next, the three-dimensional model of the target face can be reconstructed through the identity coefficient and the expression coefficient of the face by the 3DMM model, and the Z-axis of the model is further screened to obtain the depth data of the reconstructed target face. The data can be consistent with the depth data of the to-be-selected four-channel image. The supervision can be represented as:

[0159] L depth = f (‖D depth -D z ‖ 2 )

[0160] Wherein f (‖D depth -D z || 2 ) represents the Euclidean distance of the point cloud mean, and the point cloud can be understood as the three-dimensional model of the target face reconstructed through the identity coefficient and the expression coefficient of the face by the 3DMM model. D depth represents the depth information of the corresponding image under the view angle, that is, the depth data of the target four-channel, and D z represents the depth data of the reconstructed target face.

[0161] Then, the point cloud nearest neighbor distance is used to calculate the distance between the point in the point cloud and the nearest neighbor point in another point cloud between any two different view angles of the three view angles. The point cloud nearest neighbor distance loss function can be recorded as:

[0162]

[0163] Wherein N represents the number of points in the point cloud, x i represents a point at the current time, and y represents a point under another view angle through pose change to the point cloud under the current view angle. Through the constraint, the convolutional neural network can learn more fully the multi-view face information.

[0164] The image consistency loss function can be recorded as:

[0165]

[0166] Wherein I i (x) represents the target four-channel image of the i-th view angle, and I′ i (x) represents the reconstructed image corresponding to the target four-channel image obtained by reconstructing the i-th view angle.

[0167] The face point loss function can be recorded as:

[0168]

[0169] wherein q n (x) represents the face key points selected from the reconstructed three-dimensional face according to the correspondence between the key point indexes in the target four-channel image. q n represents the corresponding face key points in the to-be-selected four-channel image. Because the reconstructed three-dimensional key points and the two-dimensional key points in the to-be-selected four-channel image are not semantically consistent, the face point loss function can be calculated in two parts. The left half part only calculates the unoccluded part, and Vn represents the number of unoccluded face key points. The right half part is used to calculate the occluded part, q i represents the two-dimensional key points that are not semantically consistent with the three-dimensional key points, and y represents all the points after the projection of the reconstructed three-dimensional key points. Wherein q n , q i The face can be divided by the orientation. When the target four-channel image is to the right, it is considered that all the key points on the left half are visible. When the target four-channel image is to the left, it is considered that all the key points on the right half are visible.

[0170] Finally, according to the above parameters, the following objective function can be used for optimization:

[0171] LOSS=αL ept +βL depth +χL CD +δL photo +εL landmark

[0172] Wherein, α, β, χ, δ, ε represent the adjustment coefficients of each loss function. Through the loss function, the gradient descent method is used to train the initial convolutional neural network, and multiple iterations are performed until the parameters converge, so that the trained convolutional neural network can be obtained.

[0173] In the embodiment, the initial convolutional neural network is trained by using the objective function, and the trained convolutional neural network is obtained. The objective function fully utilizes the richness of the three-dimensional face information carried by the multi-view data and the relationship between the multi-view data in the three-dimensional space and after imaging, and improves the accuracy of the three-dimensional face reconstruction model.

[0174] As Figure 8 shown, the embodiment of the application further provides a three-dimensional face reconstruction device 10, which comprises:

[0175] An acquisition module 1001 is configured to acquire a plurality of target four-channel images corresponding to a target face, and the target four-channel images have different viewing angles.

[0176] The processing module 1002 is configured to input the plurality of target four-channel images into the pre-trained convolutional neural network to obtain an initial face three-dimensional model and expression coefficients and pose coefficients of the target face, and the initial face three-dimensional model is obtained by fusing a three-dimensional average face model and the expression coefficients.

[0177] The determining module 1003 is configured to determine texture coefficients of the target face according to the plurality of target four-channel images.

[0178] The fusion module 1004 is configured to fuse the pose coefficients and the texture coefficients with the face three-dimensional model to obtain a three-dimensional face corresponding to the target face.

[0179] The processing module 1002 is specifically configured to input the plurality of target four-channel images into the convolutional neural network to obtain intermediate layer features of the target four-channel images; splice the intermediate layer features to obtain spliced intermediate layer features; and input the spliced intermediate layer features into a model subnetwork of the convolutional neural network to obtain the initial face three-dimensional model and the expression coefficients; and input the intermediate layer features into a pose regression subnetwork of the convolutional neural network to obtain the pose coefficients.

[0180] The determining module 1003 is specifically configured to respectively establish a Gaussian pyramid model, a Laplacian pyramid model and a mask pyramid model according to the target four-channel images; and determine the texture coefficients according to the Gaussian pyramid model, the Laplacian pyramid model and the mask pyramid model.

[0181] The determining module 1003 is specifically configured to generate a hybrid Gaussian pyramid according to the Gaussian pyramid and the mask pyramid; generate a hybrid Laplacian pyramid according to the Laplacian pyramid and the mask pyramid; add and up-sample the hybrid Gaussian pyramid and the hybrid Laplacian pyramid to obtain a sampling pyramid; and add the sampling pyramid and the Laplacian pyramid to determine the texture coefficients.

[0182] The obtaining module 1001 is specifically configured to obtain a video sequence corresponding to the target face, the video sequence including a plurality of candidate four-channel images corresponding to the target face; determine pose data of each candidate four-channel image, the pose data including a pitch angle, a yaw angle and a roll angle; and select the plurality of target four-channel images from the plurality of candidate four-channel images according to the pose data of each candidate four-channel image and preset pose combination information.

[0183] The acquisition module 1001 is specifically further configured to perform key point detection on the first candidate four-channel image to obtain a first point set, wherein the first candidate four-channel image is any candidate four-channel image in the plurality of candidate four-channel images; take a set of key points corresponding to a three-dimensional average face model as a second point set, and determine a correspondence relationship between each first point set and the second point set; pair the first point set and the second point set to obtain a plurality of point set pairs, each point set pair including one key point in the first point set and one key point in the second point set; determine dimension transformation information corresponding to each point set pair respectively; and determine pose data of the first candidate four-channel image according to the dimension transformation information corresponding to each point set pair.

[0184] The training module 1005 is configured to train the initial convolutional neural network based on a target loss function to obtain the convolutional neural network, wherein the target loss function includes at least one of a key point to epipolar line distance loss function, a depth data loss function, a point cloud nearest neighbor distance loss function, an image consistency loss function, and a face point loss function.

[0185] Referring to Figure 9 The embodiment also provides a computer device, which comprises a processor 2001, a memory 2002, and a bus, the memory 2002 stores machine readable instructions executable by the processor 2001, when the computer device is running, the machine readable instructions are executed, the processor 2001 and the memory 2002 communicate through the bus, and the processor 2001 is configured to execute the steps of the three-dimensional face reconstruction method in the above embodiment.

[0186] The memory 2002, the processor 2001, and the bus are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, the elements can be electrically connected to each other through one or more communication buses or signal lines. The data processing apparatus of the three-dimensional face reconstruction system comprises at least one software functional module stored in the memory 2002 in the form of software or firmware or solidified in an operating system (OS) of the computer device. The processor 2001 is configured to execute the executable modules stored in the memory 2002, such as the software functional modules and computer programs included in the data processing apparatus of the three-dimensional face reconstruction system.

[0187] The memory 2002 can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), and the like.

[0188] Optionally, the present application also provides a storage medium, and the storage medium stores a computer program. When the computer program is run by a processor, the steps of the above method embodiments are executed. The specific implementation and technical effects are similar, and will not be repeated here.

[0189] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working process of the above-described system and device can refer to the corresponding process in the method embodiment, and will not be repeated in the present application. In the several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented by other means. The device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and actual implementation can have another division manner, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be indirect coupling or communication connection through some communication interface, device or module, which can be electrical, mechanical or other forms.

[0190] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. When the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or say the parts that make contributions to the prior art or parts of the technical solutions can be embodied in the form of software products, and the computer software products are stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk, and various storage program codes.

[0191] The above description is provided as an enabling teaching of the application and is not intended to limit its scope in any way. Any modification of the application in keeping with the spirit thereof that is apparent to those skilled in the art is to be considered within the scope of the application as defined by the appended claims.

Claims

1. A three-dimensional face reconstruction method, characterized in that, The method comprises: obtaining a plurality of target four-channel images corresponding to a target face, the target four-channel images being different in perspective; inputting the plurality of target four-channel images into a pre-trained convolutional neural network to obtain an initial three-dimensional face model, expression coefficients and pose coefficients of the target face, the initial three-dimensional face model being obtained by fusing a three-dimensional average face model and the expression coefficients; wherein the expression coefficients are used to describe the change of one or more states of facial muscles; the pose coefficients are used to describe the positional deviation of the target face relative to a standard angle, the standard angle being the pose of the three-dimensional average face model; determining texture coefficients of the target face according to the plurality of target four-channel images; fusing the pose coefficients and the texture coefficients with the initial three-dimensional face model to obtain a three-dimensional face corresponding to the target face; the determining of the texture coefficients of the target face according to the plurality of target four-channel images comprises: establishing a Gaussian pyramid model, a Laplacian pyramid model and a mask pyramid model respectively according to each of the target four-channel images; wherein the Gaussian pyramid model is established for a left face, the Laplacian pyramid model is established for a right face, and the mask pyramid model is established for a front face; determining the texture coefficients according to the Gaussian pyramid model, the Laplacian pyramid model and the mask pyramid model.

2. The three-dimensional face reconstruction method of claim 1, wherein, the inputting of the plurality of target four-channel images into the pre-trained convolutional neural network to obtain the initial three-dimensional face model, the expression coefficients and the pose coefficients of the target face comprises: inputting the plurality of target four-channel images into the convolutional neural network to obtain intermediate layer features of each of the target four-channel images; splicing each intermediate layer feature to obtain spliced intermediate layer features; inputting the spliced intermediate layer features into a model subnetwork of the convolutional neural network to obtain the initial three-dimensional face model and the expression coefficients; inputting each intermediate layer feature into a pose regression subnetwork of the convolutional neural network to obtain the pose coefficients.

3. The three-dimensional face reconstruction method of claim 1, wherein, the determining of the texture coefficients according to the Gaussian pyramid model, the Laplacian pyramid model and the mask pyramid model comprises: generating a hybrid Gaussian pyramid according to the Gaussian pyramid and the mask pyramid; generating a hybrid Laplacian pyramid according to the Laplacian pyramid and the mask pyramid; adding and up-sampling the hybrid Gaussian pyramid and the hybrid Laplacian pyramid to obtain a sampling pyramid; adding the sampling pyramid and the Laplacian pyramid to determine the texture coefficients.

4. The three-dimensional face reconstruction method of any one of claims 1-3, wherein, the obtaining of the plurality of target four-channel images corresponding to the target face comprises: obtaining a video sequence corresponding to the target face, the video sequence comprising a plurality of candidate four-channel images corresponding to the target face; determining pose data of each candidate four-channel image, the pose data comprising a pitch angle, a yaw angle and a roll angle; Based on the pose data of each candidate four-channel image and the preset pose combination information, the multiple target four-channel images are selected from the multiple candidate four-channel images.

5. The three-dimensional face reconstruction method of claim 4, wherein, The determination of pose data for each candidate four-channel image includes: Key point detection is performed on the first candidate four-channel image to obtain a first point set, wherein the first candidate four-channel image is any one of the multiple candidate four-channel images; The set of key points corresponding to the three-dimensional average face model is used as the second point set, and the correspondence between each first point set and the second point set is determined. Pair the first point set and the second point set to obtain multiple point set pairs, each point set pair including a key point in the first point set and a key point in the second point set; Determine the dimensional transformation information corresponding to each point set pair; Based on the dimensional transformation information corresponding to each point set, the pose data of the first candidate four-channel image is determined.

6. The three-dimensional face reconstruction method of any one of claims 1-3, wherein, The method further includes: The initial convolutional neural network is trained based on the target loss function to obtain the convolutional neural network. The target loss function includes at least one of the following: key point to epipolar distance loss function, depth data loss function, point cloud nearest neighbor distance loss function, image consistency loss function, and face point loss function.

7. A three-dimensional face reconstruction apparatus, characterized by comprising: The device includes: The acquisition module is used to acquire multiple target four-channel images corresponding to the target face, wherein the viewpoints of the target four-channel images are different; The processing module is used to input the multiple target four-channel images into a pre-trained convolutional neural network to obtain an initial 3D face model and the expression coefficients and pose coefficients of the target face. The initial 3D face model is obtained by fusing a 3D average face model with the expression coefficients. The expression coefficients are used to describe the changes in one or more states of the facial muscles. The pose coefficients are used to describe the orientation deviation of the target face relative to a standard angle, where the standard angle is the pose of the 3D average face model. The determination module is used to determine the texture coefficients of the target face based on the multiple target four-channel images; The fusion module is used to fuse the pose coefficients and texture coefficients with the initial 3D face model to obtain the 3D face corresponding to the target face; The determining module is specifically used to establish a Gaussian pyramid model, a Laplacian pyramid model, and a mask pyramid model based on each of the target four-channel images; wherein, the Gaussian pyramid model is established for the left side of the face, the Laplacian pyramid model is established for the right side of the face, and the mask pyramid model is established for the frontal face; and the texture coefficients are determined based on the Gaussian pyramid model, the Laplacian pyramid model, and the mask pyramid model.

8. A computer device, comprising: The computer device comprises a processor, a storage medium and a bus, the storage medium stores machine readable instructions executable by the processor, when the computer device is running, the processor communicates with the storage medium through the bus, and the processor executes the machine readable instructions to perform the steps of the three-dimensional face reconstruction method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the three-dimensional face reconstruction method according to any one of claims 1-6.

Images