A method for constructing large-scale diverse face image and model data pairs

Abstract

The present invention discloses a method for constructing large-scale diversified face images and model data pairs. The steps are: fitting the parameters of a three-dimensional deformation model for any face image to obtain a bilinear model representing the three-dimensional shape of the face; randomly selecting a suitable texture map from a face database according to the age range and gender of the acquired image; rendering the fitted face model and the selected texture map according to the original image posture under uniform illumination to generate a face image consistent with the model; training a completion network to complete the completion of the missing area in the rendered image; performing semantic segmentation on the original acquired face image to obtain the facial area of ​​the original image; adjusting the rendered image after completion to be consistent with the facial area of ​​the original image; and finally performing Poisson fusion in the facial area to obtain a face image in which the facial area is highly corresponding to the three-dimensional model. The method of the present invention can obtain large-scale diversified face images and corresponding true value three-dimensional model data pairs.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a method for constructing large-scale diversified face image and model data pairs. Background Art

[0002] In recent years, computer vision technology has developed rapidly, and 3D face models have been widely used in film and television animation, virtual reality, game design, human-computer interaction and other fields. A large number of end-to-end single-view face 3D reconstruction algorithms based on deep learning have been proposed. However, learning-based methods require a large number of high-precision face images and corresponding 3D true value data. At the same time, in order to ensure the generalization of the model, the training data should also have a high degree of realism. At present, in the field of single-view face 3D reconstruction, there is still a significant gap between the reconstructed geometric model and the true value. The reason is that in terms of data, the quality of the data set restricts the reconstruction accuracy of the supervised training predictor.

[0003] There are currently three main types of data sets used for face reconstruction.

[0004] The first category is to use the 3D face model obtained by original scanning or multi-view 3D reconstruction in laboratory scenes as the 3D truth value. This type of data has high accuracy, but is expensive. In addition, the scale of this type of data set is usually small, and the experimental scene, lighting and other conditions of the original collected pictures or videos are relatively simple, which generally cannot meet the needs of training neural networks. For example, the MICC data set proposed by Bagdanov and Andrew D et al. (Bagdanov AD, Del Bimbo A, Masi I. The florence 2d / 3d hybrid face dataset [C] / / Proceedings of the 2011 joint ACM workshop on Humangesture and behavior understanding. 2011: 79-80.) has high face accuracy, but the data volume is small.

[0005] The second category is to render two-dimensional face images from the acquired three-dimensional face model to obtain a synthetic data set. Compared with the first category, this type of data can increase the number of two-dimensional face images, and can render two-dimensional face images of any posture as needed, and the data is easy to obtain. For example, Richardson et al. (Richardson E, Sela M, Kimmel R. 3D face reconstruction by learning from synthetic data [C] / / 2016 fourth international conference on 3D vision (3DV). IEEE, 2016: 460-469.) obtained two-dimensional face images from the Bosphorus data set by weak perspective projection rendering to form a three-dimensional face reconstruction data set. However, the images rendered and synthesized by this type of data set are quite different from the real images, lacking information such as hair background, and have poor realism.

[0006] The third widely used single-view face 3D reconstruction dataset is to fit the face 3D model from the 2D face image dataset using optimization methods, and the image and the fitted 3D model are regarded as a set of data pairs. For example, the 300W-LP dataset used in the work of Xiangyu Zhu et al. (Zhu X, Lei Z, Liu X, et al. Face alignment across large poses: A 3d solution [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 146-155.) is a large dataset with a variety of face images. However, the image resolution of this dataset is not high, and the fitting process mainly relies on sparse face key points as constraints. Therefore, there is no strict correspondence between the 3D face and the face image, and there is a deviation. The accuracy of this type of data is low.

[0007] Therefore, in the field of single-view face 3D reconstruction, it is necessary to study a method that can generate a large number of face images and 3D face data pairs with high correspondence and good generalization. Summary of the invention

[0008] In view of the defects existing in the above-mentioned existing data / data generation methods, the purpose of the present invention is to provide a method that can generate a large number of face image data sets with accurate three-dimensional truth value annotations and strong sense of reality. The three-dimensional face model generated by this method has a high correspondence with the face image, and can be used for single-view face three-dimensional reconstruction tasks to improve generalization ability and reconstruction accuracy.

[0009] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the method of the present invention is as follows:

[0010] A method for constructing large-scale diversified face image and model data pairs, comprising the following steps:

[0011] S1, fitting the parameters of the 3D deformable model to any face image in the large-scale 2D face image dataset, and obtaining a bilinear model representing the geometric shape of the face as the 3D truth value;

[0012] S2, estimating the age and gender corresponding to the face image in step S1, and randomly selecting a texture map corresponding to the age and gender of the face image from the face database;

[0013] S3, rendering the bilinear model of the texture map obtained: using the geometric model fitted in step S1 and the texture map obtained in step S2 to render under uniform lighting conditions to obtain a face image consistent with the three-dimensional model;

[0014] S4, training a missing completion network, using the missing completion network to complete the missing area in the face image rendered in step S3, while other pixels on the face remain unchanged;

[0015] S5, performing semantic segmentation on the face image in step S1 through a face semantic segmentation network, or directly using the corresponding semantic segmentation value for the face image with a semantic segmentation true value to obtain the corresponding face area;

[0016] S6, training a facial edge skin completion network, using the skin completion network to complete the missing skin area of ​​the rendered image after the missing area is completed in step S4 compared with the facial area of ​​the face image in the original step S1, if the facial area of ​​the face image in the original step S1 exceeds the rendered image after completion, deforming the facial area of ​​the exceeding part to the intersection area of ​​the two, so that the rendered image after completion is completely consistent with the facial area of ​​the face image in the original step S1;

[0017] S7, using the rendered image with consistent facial region obtained in step S6 and the deformed image of the facial edge deformation of the original face image in step S1 to perform Poisson fusion replacement in the facial region, to obtain a nearly unconstrained face image with a high degree of correspondence between the face and the three-dimensional model;

[0018] S8, repeating steps S1 to S7 for different face images, thereby obtaining large-scale and diverse face images and corresponding true-value three-dimensional model data pairs.

[0019] Furthermore, the specific method of step S1 is: detect the two-dimensional key points of the input image, randomly select a three-dimensional deformable model from the three-dimensional deformable model of the Eastern or Western face, alternately optimize the identity coefficient and the expression coefficient of the bilinear model for the key points, obtain the coefficient of the bilinear model representing the geometric shape of the Eastern or Western face, and then obtain the three-dimensional face model corresponding to the image.

[0020] Furthermore, the specific method of step S2 is: for the face picture selected in step S1, if the face picture has true age and gender annotations, use the annotation values; otherwise, estimate the age and gender corresponding to the face picture, and according to the person type of the three-dimensional deformable model of the face selected in step S1, randomly select a texture map of the same type of person who is suitable for the age and gender of the picture from the Eastern or Western face database.

[0021] Furthermore, in step S4, the missing completion network is implemented by a generative network, and the training data preparation process is as follows: the facial region F of the parameterized model with consistent topology in the face data set and the high-precision model Raw collected originally are used as a set of data, and a set of two-dimensional face images are rendered under the same conditions of illumination, viewing angle, projection model, etc., respectively, and recorded as I F and I Raw ; Among them, picture I F The face has missing areas; considering the image I Raw The area contained is larger than image I F , Picture I F The area within the outer contour is set to 1 to obtain a binary mask template M. Take picture I F and I Raw′ As paired images, I F As input, I Raw′ as a truth value.

[0022] Furthermore, the specific method of step S6 is: first, remove the non-facial area that is not desired to appear in the rendered image after the missing area is completed in step S4, and take the intersection with the facial area binary mask A obtained in step S5, recorded as B; then splice A and B in the channel dimension as the input of the skin completion network; the skin completion network is implemented using a generative network, and the preparation process of its training data is as follows: segment the facial area from a large number of arbitrary images, remove facial pixels at the edge of the face with ellipses of random sizes to simulate the lack of facial skin, and splice the face with the elliptical area removed and the complete facial area binary mask A in the channel dimension to obtain the training input data; the true value data is a complete face image, and each group of input data and true value data forms a pair of data pairs for training the skin completion network.

[0023] The present invention constructs a new method for large-scale diversified face images and model data pairs. Starting from any face image in a large-scale face image data set, the method generates face image data with high correspondence and three-dimensional true value through processes such as face parameter model fitting, age and gender estimation, texture map selection, rendering, completion, and fusion face replacement. Compared with the prior art, the present invention has the following significant advantages: (1) The method has good generalization and can process any face image; (2) The correspondence between the three-dimensional model of the generated data and the face image is good, which reduces the deviation of the three-dimensional face data obtained by direct fitting; (3) The obtained data has a strong sense of reality and is used in the field of single-view face three-dimensional reconstruction, which improves the accuracy and generalization of the reconstruction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 Flow chart of the method of the present invention.

[0025] Figure 2 Flowchart of each operation stage in the embodiment of the present invention.

[0026] Figure 3 (a) a three-dimensional face model, (b) a texture map, and (c) a rendered two-dimensional face image in an embodiment of the present invention.

[0027] Figure 4 This is a training data sample of the missing completion network in an embodiment of the present invention.

[0028] Figure 5 This is a training data sample of the skin completion network in an embodiment of the present invention.

[0029] Figure 6 This is a pair of face images and three-dimensional face data generated partially in an embodiment of the present invention.

[0030] Figure 7 This is a qualitative result of an ablation experiment using the data generated by this method in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0032] like Figure 1 As shown, a method for constructing large-scale diversified face images and model data pairs in this embodiment includes the following steps:

[0033] (1) Using FFHQ (Karras T, Laine S, Aila TA style-based generator architecture for generative adversarial networks [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2019:4401-4410.), CelebA-HQ (Lee CH, Liu Z, Wu L, et al. Maskgan: Towards diverse and interactive facial image manipulation[C] / / Proceedings of the IEEE / CVF Conference onComputer Vision and Pattern Recognition.2020:5549-5558.), IMBD-WIKI(Rothe R, Timofte R, Van Gool L.Dex: Deep expectation of apparent age from a single image[C] / / Proceedings of the IEEE international conference on computer The face pictures provided in visionworkshops.2015:10-15.), using FaceScape (Yang The face bilinear model fitting algorithm provided in H, Zhu H, Wang Y, et al. Facescape: a large-scale high quality 3d face dataset and detailed riggable3d face prediction [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2020: 601-610.) fits the face shape in the picture.For each face image, use the Face alignment library (Bulat A, Tzimiropoulos G. How far are we from solving the 2d&3d face alignment problem? (and a dataset of 230,000 3d facial landmarks) [C] / / Proceedings of the IEEE International Conference on Computer Vision. 2017: 1021-1030.) to detect each image I. src N (N = 68) facial key points l = [l1, l2, ..., l N ], let the camera projection matrix be P, then the least squares method can be used to solve the corresponding positions of N facial key points in the world coordinate system The geometry of the parametric face model is given by the identity coefficients and expression coefficient Control, denoted as The parameterized model has a consistent topology, and the key points of the 3D face on the model are Identity coefficient and expression coefficient It can be solved by minimizing the energy equation E:

[0034]

[0035] Where s is the scale factor, R is the rotation matrix, and t is the translation vector. In order to increase the diversity of the three-dimensional data, a parameterized model with oriental attributes (FaceScape) or western attributes (HeadSpace) is randomly selected for each image.

[0036] (2) In order to ensure the quality and realism of the final generated image, it is necessary to select a texture map that is suitable for the age and gender of the original image for the fitted face model. The age and gender recognition network (AgeGender) is used to obtain the estimated value of the age and gender of the image, and the consistent texture maps are randomly selected from the Eastern and Western face databases. The Eastern face texture map database is generated using the FaceScape dataset, and the Western face texture map database is generated using the HeadSpace dataset.

[0037] (3) A set of corresponding face 3D models and texture maps obtained in steps (1) and (2) are rendered under uniform lighting conditions. The rendering process is implemented using the pyrender library, and the projection mode is set to weak perspective projection. Figure 3(a) is the fitted 3D face model, (b) is the selected corresponding texture map, and (c) is the result of model rendering.

[0038] (4) Figure 3 As can be seen from Figure (c), since the parametric model of the face does not include eyes, teeth, etc., the rendered image lacks the positions of eyes, teeth, etc. The pix2pixHD network (Wang TC, Liu MY, Zhu JY, et al. High-resolution image synthesis and semantic manipulation with conditional gans [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 8798-8807.) is used to complete the areas such as eyes and teeth. The training data preparation process is as follows: The facial area of ​​the topologically consistent parametric model in the face dataset (denoted as F) and the original high-precision model (denoted as Raw) are used as a set of data, and a set of two-dimensional face images are rendered under the same lighting, viewing angle, projection model, etc., denoted as I F and I Raw . Among them I F And render to get the picture Figure 3 (c) is similar, but lacks areas such as eyes and teeth. Raw The image is rendered from the original high-precision model, including areas such as eyes and teeth. Raw The area contained is greater than I F , I F The area within the outer contour is set to 1 (the rest of the area is 0) to obtain a binary mask template M. Take I F and I Raw′ As paired images to train the missing completion network, I F As input, I Raw′ As the true value, a set of training data samples such as Figure 4 As shown, (a) Figure I F , (b) Figure I Raw′ The loss function of the missing completion network during training is:

[0039]

[0040]

[0041] Among them, G represents the generator network, D represents the discriminator network, The definition is as follows:

[0042]

[0043] Where V represents the total number of pixels in the visible area of ​​the true image, and I gt represents the true value image, I x Represents the image generated by the network. During training, take λ adv =1,λ pixel =10.

[0044] (5) For the original image I src BiSeNet (Yu C, Wang J, Peng C, et al. Bisenet: Bilateral segmentation network for real-time semantic segmentation [C] / / Proceedings of the European conference on computer vision (ECCV). 2018: 325-341.) is used for semantic segmentation, and the facial region M is obtained according to the semantic segmentation map. face .

[0045] (6) First, remove the unwanted red / blue headgear in the rendered image after the missing parts are completed in step (4), and take the intersection with the binary mask of the facial area obtained in step (5) (denoted as A), denoted as B. Then concatenate A and B in the channel dimension as the input of the skin completion network. The skin completion network is implemented using the pix2pixHD network. The training data is prepared as follows: the facial area is segmented from more than 90,000 randomly taken pictures, and the facial pixels are removed at the edge of the face with a randomly sized ellipse to simulate the missing facial skin. The face in the removed elliptical area ( Figure 5 (a)) and the complete face region binary mask ( Figure 5 (b)) are spliced ​​in the channel dimension to obtain the input data for training; the true value data is the complete face image ( Figure 5 A set of training data samples is as follows: Figure 5 As shown. For each set of input data and true value data, a pair of data is formed for training the skin completion network. At the same time, the original image I src If the facial area exceeds the part of the completed rendering image, the facial area that exceeds the part is deformed to the intersection of the two, so that the facial areas of the two are completely consistent. The loss function of the skin completion network during training is:

[0046]

[0047]

[0048] in Use VGG-19 network to implement. During training, take λ rec =0.5,λ GAN =0.1,λ perc =1,λ pixel =1.

[0049] (7) Using the complementary rendering image consistent with the facial region obtained in step (6) and the deformed image of the facial edge deformation of the original image, Poisson fusion replacement is performed in the facial region to obtain a nearly unconstrained face image with a high degree of correspondence between the face and the three-dimensional model. Figure 6 Some highly corresponding face images and three-dimensional face data pairs generated by this method are shown. The first row is the face images, and the second row is the result of alignment between the three-dimensional true values ​​and the images. It can be seen that the face images have a strong sense of reality, and the correspondence between the three-dimensional model of the generated data and the face images is good, which reduces the deviation of the three-dimensional face data obtained by direct fitting.

[0050] (8) Repeat steps (1) to (7) for randomly selected images from FFHQ, CelebA, and IMBD-WIKI to obtain 100,000 sets of diverse face images and corresponding ground-truth 3D model data pairs. To demonstrate the effectiveness of the method of the present invention, the following experiments were conducted:

[0051] An ablation experiment was conducted on PRNet (Feng Y, Wu F, Shao X, et al. Joint 3D face reconstruction and dense alignment with position map regression network [C] / / Proceedings of the European conference on computer vision (ECCV). 2018: 534-551.), a typical method in the field of single-view face reconstruction. The experiment maintained the same experimental settings as PRNet, and only replaced the training data of PRNet with the data generated by the present invention. The quantitative results are as follows:

[0052] Table 1: Quantitative results of ablation experiments

[0053]

[0054] CD stands for chamfer distance, which is the average value of the distance from the vertex to the facet between the predicted 3D model and the true 3D model. It measures the overall error distance of the predicted 3D model and is defined as follows:

[0055]

[0056] Where Np ,N g are the number of vertices on the prediction model and the true value model, respectively. They are the prediction model and the true value model respectively. MNE stands for mean normal error, which is the average of the angles between the surface normal of the prediction model and the surface normal of the true value model. It measures the error of the local details of the model and is defined as follows:

[0057]

[0058] Where N n Represents the total number of pixels in the visible part of the two normal maps. Some qualitative visualization results such as Figure 7 shown.

[0059] By comparing the experimental results, the face images generated in the present invention have good correspondence with the three-dimensional face data, and the face images have a strong sense of reality, which reduces the data deviation caused by directly fitting the three-dimensional face data, and makes up for the defects of small data, low precision and poor sense of reality in the field of single-view face three-dimensional reconstruction. The data of this embodiment is used in the field of single-view face three-dimensional reconstruction, which significantly improves the accuracy of the face reconstruction model.

Claims

1. A method for constructing large-scale diversified face image and model data pairs, characterized in that: The steps include: S1, fitting the parameters of the 3D deformable model to any face image in a large-scale 2D face image dataset, and obtaining a bilinear model representing the geometric shape of the face as the 3D truth value; the specific method is: detecting the 2D key points of the input image, randomly selecting a 3D deformable model from the 3D deformable models of Eastern or Western faces, alternately optimizing the identity coefficients and expression coefficients of the bilinear model for the key points, obtaining the coefficients of the bilinear model representing the geometric shape of Eastern or Western faces, and then obtaining the 3D face model corresponding to the image; S2, estimating the age and gender corresponding to the face image in step S1, and randomly selecting a texture map corresponding to the age and gender of the face image from the face database; S3, rendering the bilinear model of the texture map obtained: using the geometric model fitted in step S1 and the texture map obtained in step S2 to render under uniform lighting conditions to obtain a face image consistent with the three-dimensional model; S4, training a missing completion network, using the missing completion network to complete the missing area in the face image rendered in step S3, while other pixels of the face remain unchanged; the missing completion network is implemented using a generative network, and its training data preparation process is as follows: the facial area F of the topologically consistent parameterized model in the face data set and the original high-precision model Raw are used as a set of data, and a set of two-dimensional face images are rendered under the same lighting, viewing angle, projection model and other conditions, respectively, denoted as I F and I Raw ;in Picture I F The face has missing areas; considering the image I Raw The area contained is larger than image I F , Picture I F The area within the outer contour is set to 1 to obtain a binary mask template M. Take picture I F and I Raw′ As paired images, I F As input, I Raw′ as truth value; S5, performing semantic segmentation on the face image in step S1 through a face semantic segmentation network, or directly using the corresponding semantic segmentation value for the face image with a semantic segmentation true value to obtain the corresponding face area; S6, training a facial edge skin completion network, using the skin completion network to complete the skin area missing from the rendered image after the missing area is completed in step S4 compared to the facial area of ​​the face image in the original step S1. If the facial area of ​​the face image in the original step S1 exceeds the rendered image after completion, the facial area of ​​the exceeding part is deformed toward the intersection area of ​​the two, so that the rendered image after completion is completely consistent with the facial area of ​​the face image in the original step S1. The specific method is: first, remove the non-facial area that is not desired to appear in the rendered image after the missing area is completed in step S4, and take the intersection with the facial area binary mask A obtained in step S5, recorded as B; then splice A and B in the channel dimension as the input of the skin completion network; the skin completion network is implemented using a generative network, and the preparation process of its training data is as follows: segment the facial area from a large number of arbitrary images, remove the facial pixels at the facial edge with a randomly sized ellipse to simulate the lack of facial skin, and splice the face with the ellipse area removed and the complete facial area binary mask A in the channel dimension to obtain the input data for training; The true value data is a complete face picture, and each set of input data and the true value data forms a data pair for training the skin completion network; S7, using the rendered image with consistent facial region obtained in step S6 and the deformed image of the facial edge deformation of the original face image in step S1 to perform Poisson fusion replacement in the facial region, to obtain a nearly unconstrained face image with a high degree of correspondence between the face and the three-dimensional model; S8, repeating steps S1 to S7 for different face images, thereby obtaining large-scale and diverse face images and corresponding true-value three-dimensional model data pairs.

2. A method for constructing large-scale diversified face image and model data pairs according to claim 1, characterized in that: The specific method of step S2 is: for the face picture selected in step S1, if the face picture has true age and gender annotations, use the annotation values; otherwise, estimate the age and gender corresponding to the face picture, and according to the person type of the three-dimensional deformable model of the face selected in step S1, randomly select a texture map of the same type of person with the age and gender of the picture from the Eastern or Western face database.

Images