A high-fidelity three-dimensional face model generation method based on natural text description

Abstract

The application discloses a high-fidelity three-dimensional face model generation method based on natural text description. The method comprises the following steps: based on a three-dimensional face data set, designing a face description questionnaire and manually labeling, and generating a description sentence according to a fixed sentence pattern; topologically unifying a three-dimensional face model, establishing a 3DMM model by using principal component analysis, and generating a texture map by using texture remapping; training a text analysis network by using one-hot encoding of face description and text embedding after CLIP model coding, inputting a text, and outputting predicted shape and texture one-hot encoding; training a shape prediction network and a texture generation network respectively to generate a three-dimensional face model with texture; and optimizing 3DMM coefficients and texture maps to make the face model conform to the abstract description and generate a final result. The application constructs a large-scale text-three-dimensional face data set, uses a neural network to predict 3DMM coefficients and texture maps, and then generates a three-dimensional face model conforming to the description from natural text.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision, and particularly relates to a high-fidelity three-dimensional face model generation method based on natural text description. BACKGROUND

[0002] 3D face has high requirements in many real fields, such as digital people, telepresence and movie special effects, etc. However, it is very complex to create a high-fidelity 3D face, which requires experienced modelers to spend a lot of time. In recent years, a large number of scholars have devoted themselves to text-to-image and image-to-3D synthesis, but this mode will cause a large intermediate error, and also lacks the ability to synthesize 3D face under the condition of given abstract description.

[0003] Currently, some scholars have attempted to recover three-dimensional shapes from text, but they cannot generate three-dimensional models that fully comply with the description. Chen et al. (Kevin Chen, Christopher B Choy, Manolis Savva, Angel X Chang, Thomas Funkhouser, and Silvio Savarese. Text2shape: Generating shapes from natural language by learning joint embeddings. In ACCV, pages 100-116. Springer, 2018.2) proposed generating colored 3D shapes from natural language by learning implicit cross-modal connections between the language and physical attributes of 3D shapes. In further research, Liu et al. (Zhengzhe Liu, Yi Wang, Xiaojuan Qi, and Chi-Wing Fu. Towards implicit text-guided 3d shape generation. In CVPR, pages 17896-17906, 2022.2) proposed decoupling shape and color prediction of learned features in text and shape, and proposed a word-level spatial transformer to associate word features in text with spatial features in shape. In subsequent research, CLIP (Contrastive Language-Image Pre-training) played an important role, which is a large pre-trained visual language model, and uses prompt learning to take advantage of the powerful representation of the CLIP model.Jain et al. (Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zero-shot text-guided object generation with dream fields. In CVPR, pages 867-876, 2022.2) proposed to combine neural rendering with multi-modal image and text representations to synthesize different 3D objects from natural language descriptions, Poole et al. (Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988, 2022.2) further utilized a pre-trained 2D text-to-image diffusion model and NeRF to perform text-to-3D synthesis with more flexibility.

[0004] The above studies are dedicated to recovering three-dimensional shapes from text rather than three-dimensional faces, but so far only Canfes et al. (Zehranaz Canfes, M Furkan Atasoy, Alara Dirik, and Pinar Yanardag. Text and image guided 3d avatar generation and manipulation. arXiv preprint arXiv:2202.06079, 2022.2, 7) have attempted to generate 3D faces, but their model relies on an unconstrained initial 3D face and is only applicable to short phrases. Therefore, it is still a problem to be solved to achieve fine-grained and high-quality 3D face generation from natural language text using facial prior knowledge. SUMMARY

[0005] In order to directly generate a fine three-dimensional face model from text, the present application provides a high-fidelity three-dimensional face model generation method based on natural text description.

[0006] In order to achieve the above-mentioned purposes, the technical solutions adopted by the method of the present application are as follows:

[0007] A high-fidelity three-dimensional face model generation method based on natural text description, comprising the following steps:

[0008] S1, obtain a three-dimensional face model dataset, design a face attribute description questionnaire and manually annotate, and generate a text description sentence corresponding to the three-dimensional face model based on a fixed sentence pattern to generate a face text description dataset;

[0009] S2, based on the face text description dataset, all three-dimensional face models are converted into topologically consistent models, i.e., models with the same number of vertices and face pieces, by using a non-rigid iterative nearest neighbor algorithm; texture remapping is performed to generate a texture map corresponding to the topologically consistent model; principal component analysis is used for dimensionality reduction to generate a 3DMM model, and corresponding 3DMM coefficients are obtained;

[0010] S3, according to the face data labeled in step S1, a one-hot encoding is constructed, a pre-trained visual language model is used to generate a text embedding vector, a text analysis network training data is constructed, and the text analysis network is trained;

[0011] S4, according to the one-hot encoding of step S3 and the 3DMM coefficients and texture map obtained in step 2, a network from one-hot encoding to 3DMM coefficients and texture map is trained, and after the training is completed, the geometric one-hot encoding and the texture one-hot encoding output by the text analysis network are respectively input into a shape prediction network and a texture generation network to generate corresponding 3DMM coefficients and texture maps;

[0012] S5, the parameter vectors of the shape space and the texture space output by the shape prediction network and the texture generation network are optimized through the loss function of the visual language model and the L2 loss function, so that they can conform to the abstract text description, and the final three-dimensional face model is generated.

[0013] Further, in step S3, a p x q dimensional matrix is constructed according to the face attribute description questionnaire, where p represents p different facial attributes, and q represents the dimension of one-hot encoding; an 8-layer multilayer perceptron is used as a text analysis network, the text analysis network takes the text embedding vector of the visual language model as input, and then outputs the predicted one-hot encoding, and cross-entropy loss is used as the loss function for training.

[0014] Further, in step S4, an 8-layer multilayer perceptron is used as a shape prediction network, and the shape prediction network takes the geometric one-hot encoding as input and then outputs the corresponding 3DMM coefficients.

[0015] The present application constructs a large-scale text-3D face dataset containing various facial attributes and free descriptions of different objects, and proposes a feasible method for generating a 3D face model from a free text description. The present application first realizes the generation of a 3D face model from a natural text description, filling the research gap in this regard. The proposed method can be widely applied in the fields of digital people, game creation, film special effects, etc., and has high practical value and development prospects. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 A flowchart of the method of the present application.

[0017] Figure 2 A schematic diagram of the face annotation questionnaire in the embodiment of the present application.

[0018] Figure 3 A schematic diagram of the specific area triple loss in the embodiment of the present application.

[0019] Figure 4 A flowchart of the running stage in the embodiment of the present application.

[0020] Figure 5 A result display diagram in the embodiment of the present application. DETAILED DESCRIPTION

[0021] To make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0022] As shown in Figure 1 The specific process of the high-fidelity 3D face model generation method based on natural text description of the present application is as follows:

[0023] 1. Design a face description annotation questionnaire, covering 25 dataset attributes from overall to local description of the face, such as five features, skin color, beard, etc. Figure 2The Facescape dataset (Yang, Haotian, et al. "Facescape: a large-scale high quality 3d face dataset and detailed riggable 3d face prediction." Proceedings of the ieee / cvf conference on computer vision and pattern recognition. 2020.) and the Headspace dataset (Pears, N.E. (Creator), Duncan, C. (Creator), Smith, W.A.P. (Contributor), Dai, H. (Contributor) (6 Jun 2018). The Headspace dataset. University of York. 10.15124 / 6efa9588-b715-44ec-b7bb-f10dff7ca93e) were chosen to contain 1627 3D face models, each of which was manually annotated to obtain all facial attributes, and combined into complete sentences using fixed sentence patterns such as "his [eyes] are [big] and [round]" and "he has [big] and [round] [eyes]" to form a text description-3D face dataset, as shown in Table 1. Figure 2

[0024] 2. All 3D face models were registered as topologically consistent 3D meshes using a non-rigid iterative closest point algorithm, each of which was composed of 26369 vertices and 52536 faces. Then, the texture was remapped to obtain a texture map corresponding to the topologically consistent model, the resolution of which was 1024*1024. In this method, the texture map was reduced to 512*512 resolution to reduce the volume of the network. Subsequently, all 3D face models were constructed into a 3D morphable model (3DMM) using principal component analysis (Blanz, Volker, and Thomas Vetter. "A morphable model for the synthesis of 3D faces." Proceedings of the 26th annual conference on Computer graphics and interactive techniques. 1999.), and the corresponding 3DMM coefficients were generated, each of which was represented by a 300-dimensional vector;

[0025] ​3. According to the previously annotated facial data, first construct a p x q dimensional matrix, where p represents p different facial attributes, and q represents the dimension of one-hot encoding. 24 facial attributes are selected in addition to ear shape, and p = 24, q = 8 (according to 8 eye types). Among them, 12 attributes in p are shape attribute descriptions (eye size, eye shape, eye spacing, eyelid, nose size, nose wing width, nose bridge height, nose base shape, mouth width, lip thickness, lip shape, face shape, face fat, ear shape), 9 attributes are texture attribute descriptions (eyebrow shape, eyebrow color, eyebrow density, pupil color, with or without beard, beard density, beard color, beard type), and 3 common attribute descriptions (race, gender and age). In this way, all attributes of the face are represented in the form of one-hot encoding. Then, using a large pre-trained visual language model CLIP (Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International Conference on Machine Learning. PMLR, 2021.), the annotated facial attributes are composed into sentences according to the above fixed sentence, and then the sentence is encoded into a 512-dimensional text embedding vector by the text encoder of CLIP. In this way, complex text semantics are converted into simple encoding and vector representation. An 8-layer multilayer perceptron is designed as a text analysis network, and the one-hot encoding and text embedding vector constructed before are used to train the neural network. Cross entropy loss is adopted, and the specific loss function is:

[0026]

[0027] where i is the index of the annotated facial attribute, j is the index of the feature option describing this attribute, y ij is the true value of one-hot encoding, is the predicted value of one-hot encoding, and logsoftmax represents the logarithmic normalization exponential function.

[0028] 4. According to the order of one-hot encoding, it is divided into shape attribute encoding and texture attribute encoding, and shape prediction network and texture generation network are trained respectively.

[0029] The shape prediction network is an 8-layer multilayer perceptron that takes one-hot encodings of shape attributes as input and outputs corresponding 3DMM coefficients. The network is trained using a weighted L1 loss function and triplet losses for different facial regions. The weighted L1 loss function segments the face into four distinct regions: facial keypoints (eyes, nose, mouth), other facial regions, and other regions excluding the eyes, nose, and mouth, including the ears and back of the head. Different weights are assigned based on the importance of each region, with a ratio of 68 facial keypoints: (eyes, nose, mouth): other facial regions: (ears and back of the head) = 16:4:3:0.

[0030]

[0031] Where α i V represents the weighting coefficients for different regions. i Represents the vertex of the i-th region. It is the vertex of the predicted i-th region.

[0032] The region-specific triplet loss constructs positive and negative example data pairs for faces based on different regions, making the network prediction results closer to positive examples and farther from negative examples, such as... Figure 3 As shown. Specifically, taking eyes as an example, when the sample has small eyes, a negative example with large eyes is selected from the database, and the network prediction result is made close to the true value of the sample while staying away from the negative example. The loss function is:

[0033]

[0034] in v is the vertex of the predicted i-th region. i It is the corresponding truth value. It is the vertex of its counterexample. m i and λ i These represent the threshold and weight of the corresponding region, respectively.

[0035] The texture generation network is a network structure based on StyleGAN (Karras, Tero, Samuli Laine, and TimoAila. "A style-based generator architecture for generative adversarial networks." Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2019.). It uses random noise z and one-hot encoding as input to the network and trains it to generate corresponding texture maps.

[0036] Based on the text analysis network trained in step 3, a piece of text description can be input to predict a 24*8 one-hot encoding matrix. The predicted encoding of the shape attribute and the encoding of the texture attribute are input into the trained shape prediction network and texture generation network respectively to obtain 3DMM coefficients and texture maps, and a three-dimensional face model conforming to the description can be generated.

[0037] 5. After step 4, a three-dimensional face model with texture conforming to the text description has been obtained. In addition, more unique features can be given to the abstract text description, such as "he looks like Tony Stark" and "she is made up". In the abstract feature generation, a differentiable renderer is used to render the three-dimensional face model generated in step 4 into a two-dimensional picture at three different angles of -30°, 0° and 30°, and a CLIP model is used to encode the abstract text description into an embedding vector. CLIP Loss is used to measure the difference between the generated three-dimensional model and the abstract text description:

[0038] L CLIP =1-<E T (t),E I (i)>

[0039] Where E T and E I represent the CLIP text encoder and image encoder respectively, t and i represent the input text description and rendered two-dimensional picture respectively, and <·,·> represents the cosine similarity.

[0040] At the same time, two regularization L2 loss functions are added to optimize the 3DMM coefficients and texture maps together, and the complete loss function is:

[0041]

[0042] Where s0 and t0 represent the initial values of the shape parameters and texture parameters from the specific generation respectively, and β1 and β2 are the corresponding weight coefficients.

[0043] Through the above steps, the present application can directly generate a high-fidelity three-dimensional face model from a piece of natural text description, and the generation result is as shown in Figure 5 .

[0044] The natural text-based high-fidelity three-dimensional face model generation method proposed by the present application realizes the direct generation of a three-dimensional face from a text, reduces the cost of obtaining a three-dimensional model, and is helpful for related research on three-dimensional face generation. The generated three-dimensional face can be applied to many fields such as film and television, games, security, etc.

Claims

1. A high-fidelity three-dimensional face model generation method based on natural text description, characterized in that, The method comprises the following steps: S1, obtaining a three-dimensional face model data set, designing a face attribute description questionnaire and manually labeling, and generating a text description sentence corresponding to the three-dimensional face model based on a fixed sentence pattern to generate a face text description data set; S2, based on the face text description data set, all three-dimensional face models are converted into topologically consistent models, i.e. models with the same number of vertices and face pieces, by using a non-rigid iterative nearest neighbor algorithm; texture remapping is performed to generate a texture map corresponding to the topologically consistent model; a 3DMM model is generated using principal component analysis to obtain corresponding 3DMM coefficients; S3, constructing a one-hot encoding according to the face data labeled in step S1, generating a text embedding vector using a pre-trained visual language model, constructing text analysis network training data, and training the text analysis network; S4, training a network from one-hot encoding to 3DMM coefficients and texture maps according to the one-hot encoding of step S3 and the 3DMM coefficients and texture maps obtained in step 2, and after training, inputting the geometric one-hot encoding and texture one-hot encoding output by the text analysis network into a shape prediction network and a texture generation network respectively to generate corresponding 3DMM coefficients and texture maps; S5, optimizing the parameter vectors of the shape space and the texture space output by the shape prediction network and the texture generation network through the loss function of the visual language model and the L2 loss function, so that they can conform to the abstract text description, and generating a final three-dimensional face model. 2.The method of claim 1, wherein, In step S1, the face attributes include 25 kinds. 3.The method of claim 1, wherein, In step S3, a p x q dimensional matrix is constructed according to the face attribute description questionnaire, where p represents p different facial attributes and q represents the dimension of one-hot encoding; an 8-layer multilayer perceptron is used as a text analysis network, which takes the text embedding vector of the visual language model as input and then outputs the predicted one-hot encoding, and cross-entropy loss is used as the loss function for training.

4. The method of claim 1, wherein, In step S4, an 8-layer multilayer perceptron is used as a shape prediction network, which takes the geometric one-hot encoding as input and then outputs the corresponding 3DMM coefficients.

5. The method of claim 1, wherein, The shape prediction network is trained using a weighted L1 loss function and a three-tuple loss for different facial regions: where v i denotes the true value of the vertex coordinates of the i-th region, α i denotes the corresponding weight.

6. The method of claim 5, wherein the method further comprises: The entire face is divided into four regions: facial key points, eyes, nose, mouth, other facial regions, and other regions excluding the above, wherein the other facial regions include facial regions excluding eyes, nose, and mouth, and the other regions include ears and the back of the head.

7. The method of claim 6, wherein the method further comprises: The corresponding weights of the four regions are 16:4:3:

0. 8.The method of claim 5, wherein, The three-tuple loss for different facial regions is based on constructing positive and negative example data pairs for the face in different regions, so that the network prediction result is close to the positive example and far from the negative example; and the loss function is: where is the predicted vertex of the i-th region, v i is the corresponding ground truth, is the vertex of its counterexample; m i and λ i denote the threshold and weight of the corresponding region, respectively. 9.The method of claim 1, wherein, In the step S5, the three-dimensional face model composed of the 3DMM coefficients and the texture map is rendered into a two-dimensional picture by using a differentiable renderer, and is encoded by using a visual language model encoder, a cosine similarity loss is calculated between the obtained image embedding and the input abstract text embedding, and two regular term losses are added to respectively constrain the shape space S and the texture space T, and a final loss function is as follows: Wherein, s0 and t0 respectively represent initial values of the shape space parameter vector and the texture space parameter vector, and β1 and β2 are corresponding weight coefficients.

Images