A method for generating blendshape of human face based on single-view 3D reconstruction

Abstract

The present invention discloses a method for generating blendshape of a face based on single-view 3D reconstruction, comprising the following steps: using a single-view 3D face reconstruction method based on deep learning, and using an existing data set, training a network that recovers the 3D structure of a face from a single RGB face image; importing the target model set into mainstream 3D animation software to synthesize blendshape. The present invention realizes the generation of the entire blendshape including various expression models based on a 2D face image, optimizes the process, avoids the cumbersome manual modeling operation, and greatly reduces the investment in manpower and material resources; the face reconstruction part of the first part of the present invention adopts an encoder-decoder structure, and uses a UV position map to represent 3D face information, so that the reconstructed face has richer detail information, avoiding the problem of missing face texture details in traditional methods.

Description

Technical Field

[0001] The present invention relates to the technical field of computer animation, and in particular to a method based on a single-view Figure 3 Blendshape generation method for face reconstruction. Background Art

[0002] With the development of computer technology, the ways of reproducing faces are becoming more and more diverse.

[0003] Structured light and binocular cameras can better reconstruct the three-dimensional structure of the face, but face reconstruction using face images with depth information is costly and not universal.

[0004] Therefore, how to restore a three-dimensional face from a single image information has high research value.

[0005] Facial blendshape animation technology is widely used in the fields of computer animation and games. It is a technology that simulates new facial expressions through expression coefficients.

[0006] The production of blendshape requires manually collecting a large number of facial expressions to generate face models or generating a large number of expression models based on a single face model, which is time-consuming and inefficient. Summary of the invention

[0007] The technical problem to be solved by the present invention is that a large number of facial expressions need to be manually collected to generate a face model through a single image and blendshape production, or a large number of expression models need to be produced based on a single face model, which is time-consuming and inefficient. Figure 3 A blendshape generation method for face reconstruction with 3D reconstruction is proposed to solve the above problems of time-consuming and inefficient.

[0008] The present invention is achieved through the following technical solutions:

[0009] A single-view Figure 3 The method for generating blendshape of a face reconstructed by dimensional reconstruction comprises the following steps:

[0010] Using single-view deep learning Figure 3 3D face reconstruction method, and use existing datasets to train a network to recover the 3D structure of a face from a single RGB face image;

[0011] Input a neutral face image and multiple expression images into the trained network, thereby obtaining multiple three-dimensional face models as a source model set;

[0012] Input the face image intended to generate blendshape into the trained network to generate the target model;

[0013] Automatic deformation transfer is realized by using a triangular mesh deformation transfer method and 68 facial key point positions of a human face obtained when reconstructing a human face based on deep learning, and the deformation from the natural expression to multiple unnatural expressions in the source model set is transferred to the target model according to the corresponding key points to generate a target model set;

[0014] The target model set is imported into mainstream 3D animation software to synthesize blendshape.

[0015] In some embodiments, the step utilizes a single-view deep learning-based Figure 3 The proposed method uses an existing dataset to train a network that can recover the 3D structure of a face from a single RGB face image. The training includes the following sub-steps:

[0016] Augment the dataset;

[0017] Preprocess the dataset;

[0018] Design encoder-decoder network structure;

[0019] Design loss function for network training.

[0020] In some embodiments, the step utilizes a single-view deep learning-based Figure 3 The proposed method uses existing datasets to train a network that can recover the 3D structure of a face from a single RGB face image, including:

[0021] The data set is preprocessed and a UV position map is generated.

[0022] In some embodiments, the step utilizes a single-view deep learning-based Figure 3 The proposed method uses an existing dataset to train a network that can recover the 3D structure of a face from a single RGB face image, including the following sub-steps:

[0023] Design the encoder-decoder structure network;

[0024] Input the two-dimensional face image into the encoder to extract the image area features;

[0025] After the decoder, the face UV position map parameters are obtained.

[0026] In some embodiments, the step utilizes a single-view deep learning-based Figure 3 The proposed method uses existing datasets to train a network that can recover the 3D structure of a face from a single RGB face image, including:

[0027] Design the loss function, the loss function is as follows:

[0028]

[0029] Among them, P(x,y) is the output result after the input image passes through the encoder-decoder structure network. is the ground truth of the UV position map obtained by preprocessing, and W(x,y) is the weight mask.

[0030] In some embodiments, the step utilizes a single-view deep learning-based Figure 3 The proposed method uses existing datasets to train a network that can recover the 3D structure of a face from a single RGB face image, including:

[0031] The augmented face image and the preprocessed UV position map are passed into the encoder-decoder structure network for training; the face image is used as input, the preprocessed UV position map is used as ground truth, the training loss value is calculated using the loss function, and the model training is continuously iterated.

[0032] In some embodiments, the step utilizes a single-view deep learning-based Figure 3 The method of face reconstruction based on the existing dataset is used to train a network to recover the 3D structure of a face from a single RGB face image. The network includes:

[0033] The encoder-decoder structure network includes an encoder module and a decoder module, and the encoder module uses a ResNeSt network; the encoder includes 1 single convolution layer and 8 split attention modules; each splitattention module may include a 1×1 convolution layer, a 2×2 average pooling layer, a SplitAttention convolution layer and a BatchNormal layer; the decoder module includes several upsampling layers.

[0034] In some embodiments, the step uses a triangular mesh deformation transfer method and 68 facial key point positions of a human face obtained when reconstructing a human face based on deep learning to realize automatic deformation transfer, and transfers the deformation from the natural expression to multiple unnatural expressions in the source model set to the target model according to the corresponding key points, generating a target model set, including:

[0035] The triangle mesh deformation transfer is based on affine transformation, deforming the triangle mesh of the source model, and transferring the deformation transformation through the correspondence between the source model and the target model, and for an affine transformation set T1+d1…T |T| +d |T| Need to meet To achieve consistency constraints, T is the triangle index set of the target mesh, T and d are the non-translation part and translation vector of the affine transformation, respectively, and p(νi) is the set of all triangles sharing vertices. On this basis, the constraint optimization problem formula of the deformation transfer target affine transformation is:

[0036] In some embodiments, the step uses a triangular mesh deformation transfer method and 68 facial key point positions of a human face obtained when reconstructing a human face based on deep learning to realize automatic deformation transfer, and transfers the deformation from the natural expression to multiple unnatural expressions in the source model set to the target model according to the corresponding key points, generating a target model set, including:

[0037] Solve the correspondence between the source model triangle mesh index set and the target model index set. Use the fixed point position to reconstruct the above constraint optimization formula. The undeformed and deformed vertices of the source model are represented by ν i and Represents, where i∈1...3, adding a vertex as the fourth undeformed point to each triangle mesh of the target model:

[0038] The non-translational part of the affine transformation is represented by the combination of undeformed and deformed vertices as Taking the known undeformed vertices as constants and T as the linear combination of the unknown deformed vertices, we can get the formula: where c is the vector containing the source transformation S, is the vector of unknown deformed vertex coordinates, A is Mapped to the sparse matrix of c. Setting the objective function gradient to 0 gives the normal equation Where A is the vertex position that depends only on the undeformed mesh, the columns correspond to the unknown deformed target vertices, and the number of columns is the sum of the number of target network vertices and the number of triangle meshes.

[0039] In some embodiments, the step uses a triangular mesh deformation transfer method and 68 facial key point positions of a human face obtained when reconstructing a human face based on deep learning to realize automatic deformation transfer, and transfers the deformation from the natural expression to multiple unnatural expressions in the source model set to the target model according to the corresponding key points, generating a target model set, including:

[0040] Automatic deformation transfer is achieved by reconstructing the face based on deep learning using the 68 facial key points of the face. Figure 3The proposed method uses an existing dataset to train a face reconstruction network that can recover the three-dimensional structure of a face from a single RGB face image. The UV position map obtained by regression stores the corresponding xyz coordinates of each pixel in RGB form. Therefore, the positions of 68 facial feature points can be directly used to calculate the mapping from the source model set to the target model through the iterative nearest point method.

[0041] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0042] The present invention realizes the generation of the entire blendshape including various expression models based on a two-dimensional face image, optimizes the process, avoids the tedious manual modeling operation, and greatly reduces the investment in manpower and material resources; the first part of the face reconstruction part of the present invention adopts an encoder-decoder structure, uses a UV position map to represent three-dimensional face information, and the reconstructed face has richer detail information, avoiding the problem of missing face texture details in traditional methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without creative work. In the drawings:

[0044] Figure 1 This is a schematic diagram of the main steps of the present invention;

[0045] Figure 2 It is a schematic diagram of the sub-step flow chart of the present invention;

[0046] Figure 3 It is a schematic diagram of the sub-step flow chart of the present invention;

[0047] Figure 4 It is the overall flow chart of the present invention. DETAILED DESCRIPTION

[0048] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.

[0049] like Figure 1 As shown, and refer to Figure 4 , a method based on single-view Figure 3 The method for generating blendshape of a face reconstructed by dimensional reconstruction comprises the following steps:

[0050] S100, using single view based on deep learning Figure 3 The proposed method uses existing datasets to train a network that can recover the 3D structure of a face from a single RGB face image.

[0051] In this step, the face network is restored through a deep learning three-dimensional face method and a data set.

[0052] S200, inputting a neutral face image and multiple expression images into the trained network, thereby obtaining multiple three-dimensional face models as a source model set.

[0053] A face picture is used as the original picture source material, and an expression picture is used as the material to form a source model set.

[0054] S300, inputting a face image intended to generate a blendshape into the trained network to generate a target model.

[0055] In this step, a model is established to facilitate subsequent processing.

[0056] S400, using a triangular mesh deformation transfer method and 68 facial key point positions obtained when reconstructing a face based on deep learning to achieve automated deformation transfer, transferring the deformation from the natural expression to multiple unnatural expressions in the source model set to the target model according to the corresponding key points, and generating a target model set.

[0057] In this step, the point positions of the face are automatically transferred to the target and a target model set is formed.

[0058] S500: Import the target model set into mainstream 3D animation software to synthesize blendshape.

[0059] In this step, the blendshape is synthesized through 3D software to achieve the effect.

[0060] like Figure 2 As shown, in some implementations, step S100 includes the following sub-steps:

[0061] Augment the dataset;

[0062] Preprocess the dataset;

[0063] Design encoder-decoder network structure;

[0064] Design loss function for network training.

[0065] In some implementations, the step S100 includes:

[0066] The data set is preprocessed and a UV position map is generated.

[0067] like Figure 3 As shown, in some implementations, the step S100 includes the following sub-steps:

[0068] S101. Design the encoder-decoder structure network;

[0069] S102, inputting the two-dimensional face image into the encoder to extract the image region features;

[0070] S103, after the decoder, obtain the face UV position map parameters.

[0071] In some implementations, the step S100 includes:

[0072] Design the loss function, the loss function is as follows:

[0073]

[0074] Among them, P(x,y) is the output result after the input image passes through the encoder-decoder structure network. is the ground truth of the UV position map obtained by preprocessing, and W(x,y) is the weight mask.

[0075] In some implementations, the step S100 includes:

[0076] The augmented face image and the preprocessed UV position map are passed into the encoder-decoder structure network for training; the face image is used as input, the preprocessed UV position map is used as ground truth, the training loss value is calculated using the loss function, and the model training is continuously iterated.

[0077] In some implementations, the step S100 includes:

[0078] The encoder-decoder structure network includes an encoder module and a decoder module, and the encoder module uses a ResNeSt network; the encoder includes 1 single convolution layer and 8 split attention modules; each splitattention module may include a 1×1 convolution layer, a 2×2 average pooling layer, a SplitAttention convolution layer and a BatchNormal layer; the decoder module includes several upsampling layers.

[0079] In some implementations, the step S400 includes:

[0080] The triangle mesh deformation transfer is based on affine transformation, deforming the triangle mesh of the source model, and transferring the deformation transformation through the correspondence between the source model and the target model, and for an affine transformation set T1+d1…T |T| +d |T| Need to meet To achieve consistency constraints, T is the triangle index set of the target mesh, T and d are the non-translation part and translation vector of the affine transformation, respectively, and p(νi) is the set of all triangles sharing vertices. On this basis, the constraint optimization problem formula of the deformation transfer target affine transformation is:

[0081] In some implementations, the step S400 includes:

[0082] Solve the correspondence between the source model triangle mesh index set and the target model index set. Use the fixed point position to reconstruct the above constraint optimization formula. The undeformed and deformed vertices of the source model are represented by νi and Represents, where i∈1...3, adding a vertex as the fourth undeformed point to each triangle mesh of the target model:

[0083] The non-translational part of the affine transformation is represented by the combination of undeformed and deformed vertices as Taking the known undeformed vertices as constants and T as the linear combination of the unknown deformed vertices, we can get the formula: where c is the vector containing the source transformation S, is the vector of unknown deformed vertex coordinates, A is Mapped to the sparse matrix of c. Setting the objective function gradient to 0 gives the normal equation Where A is the vertex position that depends only on the undeformed mesh, the columns correspond to the unknown deformed target vertices, and the number of columns is the sum of the number of target network vertices and the number of triangle meshes.

[0084] In some implementations, the step S400 includes:

[0085] Automatic deformation transfer is achieved by reconstructing the face based on deep learning using the 68 facial key points of the face. Figure 3 The proposed method uses an existing dataset to train a face reconstruction network that can recover the three-dimensional structure of a face from a single RGB face image. The UV position map obtained by regression stores the corresponding xyz coordinates of each pixel in RGB form. Therefore, the positions of 68 facial feature points can be directly used to calculate the mapping from the source model set to the target model through the iterative nearest point method.

[0086] In some implementations, S200 needs to use the network model trained by S100 to input a 2D face image to perform UV position map prediction, and obtain corresponding three-dimensional face model vertex information according to the obtained UV position map.

[0087] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for generating blendshape of human face based on single view 3D reconstruction, characterized in that: The steps include: Using a single-view 3D face reconstruction method based on deep learning and existing datasets, we train a network that can recover the 3D structure of a face from a single RGB face image. Input a neutral face image and multiple expression images into the trained network, thereby obtaining multiple three-dimensional face models as a source model set; Input the face image intended to generate blendshape into the trained network to generate the target model; Automatic deformation transfer is realized by using a triangular mesh deformation transfer method and 68 facial key point positions of a human face obtained when reconstructing a human face based on deep learning, and the deformation from the natural expression to multiple unnatural expressions in the source model set is transferred to the target model according to the corresponding key points to generate a target model set; Importing the target model set into mainstream 3D animation software to synthesize blendshape; Specifically, The step utilizes a single-view 3D face reconstruction method based on deep learning and utilizes an existing data set to train a network that recovers the 3D structure of a face from a single RGB face image. The steps include the following sub-steps: augmenting the data set; preprocessing the data set; designing an encoder-decoder network structure; designing a loss function for network training; The step utilizes a single-view 3D face reconstruction method based on deep learning and utilizes an existing data set to train a network that recovers the 3D structure of a face from a single RGB face image, including: preprocessing the data set and generating a UV position map; The step utilizes a single-view 3D face reconstruction method based on deep learning and uses an existing data set to train a network that recovers the 3D structure of a face from a single RGB face image, and includes the following sub-steps: designing an encoder-decoder structure network; inputting a 2D face image into the encoder to extract the image region features; and obtaining the face UV position map parameters through the decoder; The step utilizes a single-view 3D face reconstruction method based on deep learning and uses an existing data set to train a network that recovers the 3D structure of a face from a single RGB face image, including: designing a loss function, the loss function is as follows: ; Where P(x,y) is the output result after the input image passes through the encoder-decoder structure network, is the ground truth of the UV position map obtained by preprocessing, and W(x,y) is the weight mask; The step utilizes a single-view 3D face reconstruction method based on deep learning and an existing data set to train a network that recovers the 3D structure of a face from a single RGB face image, including: passing the augmented face image and the pre-processed UV position map into an encoder-decoder structure network for training; the face image is used as input, the pre-processed UV position map is used as ground truth, the loss function is used to calculate the training loss value, and the model training is continuously iterated; The step utilizes a single-view 3D face reconstruction method based on deep learning and utilizes an existing data set to train a network that recovers the 3D structure of a face from a single RGB face image, including: an encoder-decoder structure network comprising an encoder module and a decoder module, wherein the encoder module uses a ResNeSt network; the encoder internally comprises a single convolutional layer and 8 split attention modules; each of the split attention modules may comprise a 1×1 convolutional layer, a 2×2 average pooling layer, a SplitAttention convolutional layer, and a BatchNormal layer; the decoder module comprises several upsampling layers; The step utilizes the triangular mesh deformation transfer method and the 68 facial key point positions of the face obtained when reconstructing the face based on deep learning to realize automatic deformation transfer, and transfers the deformation from the natural expression in the source model set to multiple unnatural expressions to the target model according to the corresponding key points, and generates a target model set, including: the triangular mesh deformation transfer is based on affine transformation, the source model triangular mesh is deformed, the deformation transformation is transferred through the corresponding relationship between the source model and the target model, and the affine transformation set of a target model vertex set Need to meet To achieve consistency constraints, T is the triangle index set of the target mesh, T and d are the non-translation part and translation vector of the affine transformation, respectively, and 𝑝(𝑣𝑖) is the set of all triangles with shared vertices. On this basis, the constraint optimization problem formula of the deformation transfer target affine transformation is: ; The step uses the triangular mesh deformation transfer method and the 68 facial key point positions of the face obtained when reconstructing the face based on deep learning to realize automatic deformation transfer, and transfers the deformation from the natural expression to multiple unnatural expressions in the source model set to the target model according to the corresponding key points, and generates a target model set, including: solving the corresponding relationship between the source model triangular mesh index set and the target model index set, reconstructing the above constraint optimization formula using the fixed point position, and using the undeformed and deformed vertices of the source model respectively. and Indicates that , add a vertex as the fourth undeformed point to each triangle mesh of the target model: ; The non-translational part of the affine transformation is represented by the combination of undeformed and deformed vertices as , taking the known undeformed vertices as constants and T as the linear combination of the unknown deformed vertices, we can get the formula: , where c is the vector containing the source transformation S, is the vector of unknown deformed vertex coordinates, A is Mapped to the sparse matrix of c; setting the objective function gradient to 0 gives the normal equation , where A is the vertex position that depends only on the undeformed mesh, the columns correspond to the unknown deformed target vertices, and the number of columns is the sum of the number of target network vertices and the number of triangle meshes; The step uses a triangular mesh deformation transfer method and 68 facial key point positions of the face obtained when reconstructing the face based on deep learning to achieve automatic deformation transfer, and transfers the deformation from the natural expression to multiple non-natural expressions in the source model set to the target model according to the corresponding key points to generate a target model set, including: realizing automatic deformation transfer through the 68 facial key point positions of the face obtained when reconstructing the face based on deep learning, the step uses a single-view three-dimensional face reconstruction method based on deep learning, and uses an existing data set to train a face reconstruction network in a network that recovers the three-dimensional structure of the face from a single RGB face image. The UV position map obtained by regression stores the corresponding xyz coordinates of each pixel in RGB form, so the 68 facial feature point positions can be directly used to calculate the mapping from the source model set to the target model through an iterative nearest point method.

Images