A training method and a generating device of a geometrically more realistic three-dimensional perceptual face GAN model

Abstract

The application discloses a more geometrically realistic three-dimensional perception face GAN model, which comprises the following steps: collecting portrait images from a Flickr image website, extracting camera poses of the portrait images, filtering out small-pose images, and obtaining a large-pose face dataset LPFF; combining the FFHQ and the LPFF to obtain a dataset FFHQ+LPFF; resampling the FFHQ+LPFF to obtain a dataset FFHQ+LPFF-rebal; and training the more geometrically realistic three-dimensional perception face GAN model by using the FFHQ+LPFF and the FFHQ+LPFF-rebal datasets. FFHQ The application also discloses a more geometrically realistic three-dimensional perception face GAN model generation device. The application establishes a high-quality large-pose face dataset and trains a more geometrically realistic three-dimensional perception face GAN model, solves the problem of insufficient large-pose data in the existing face dataset, and generates face geometry conforming to the physiological structure of a real face.

Description

Technical Field

[0001] This invention relates to the fields of portrait editing and portrait generation technology, and in particular to a training method and generation device for a geometrically more realistic 3D perceptual face GAN model. Background Technology

[0002] Recently, a number of 3D perception generators have been proposed to learn how to generate consistent 3D facial portraits from 2D image datasets.

[0003] The IEEE / CVF Conference on Computer Vision and Pattern Recognition (2022), pp. 16123-16133, "Efficient geometry-aware 3D generative adversarial networks" discloses a method using generative adversarial networks and neural body rendering to synthesize vision under different camera poses. Figure 1 The result was...

[0004] However, 3D perception generators trained on existing 2D image datasets struggle to produce realistic facial geometry. This problem is exacerbated when rendering results under large poses. Faces synthesized using these methods exhibit noticeable artifacts, including distortion, blurring, and layering, all indicating that the generated face shapes are not realistic enough.

[0005] The aforementioned problems in 3D face-aware GAN models are mainly caused by the imbalanced camera pose distribution in the training dataset. The IEEE / CVF conference on computer vision and pattern recognition (2019), pp. 4401-4410, "A style-based generator architecture for generative adversarial networks," discloses the FFHQ dataset, a popular high-quality face dataset. Most current 3D face generators are trained on this dataset, but FFHQ contains very few large-pose face images. Therefore, 3D face-aware GAN models cannot learn correct large-pose faces and realistic face geometry. Summary of the Invention

[0006] To address the aforementioned issues, this invention proposes a training method for a geometrically more realistic 3D perceptive face GAN model. First, this invention designs a data collection process capable of extracting large-pose face images, establishing a high-quality large-pose face dataset. Then, based on this dataset, a geometrically more realistic 3D perceptive face GAN model is trained, thereby generating face geometry that conforms to the physiological structure of real human faces.

[0007] A training method for a geometrically more realistic 3D face-perceiving GAN model includes the following steps:

[0008] (1) Using the camera poses of the FFHQ portrait dataset, calculate the camera pose density function ρ of the FFHQ portrait dataset with respect to angles θ and φ. FFHQ ;

[0009] (2) Collect portrait images from the Flickr image website, extract the camera pose of the portrait images, filter out small pose face images, and obtain the large pose face image dataset LPFF. Combine the FFHQ dataset with the LPFF dataset to obtain the dataset FFHQ+LPFF.

[0010] (3) Resample the FFHQ+LPFF dataset to make it more balanced. The resampled dataset is denoted as FFHQ+LPFF-rebal. Calculate the camera pose density function ρ of FFHQ+LPFF with respect to angles θ and φ. FFHQ+LPFF For all images in FFHQ+LPFF, the images are repeatedly calculated based on the density of the corresponding camera pose.

[0011] (4) Use the two datasets FFHQ+LPFF and FFHQ+LPFF-rebal to train a geometrically more realistic 3D face perception GAN model.

[0012] Preferably, in step (1), the camera pose density function ρ of the FFHQ portrait dataset with respect to angles θ and φ is calculated using the camera pose of the FFHQ portrait dataset. FFHQ The specific steps are as follows:

[0013] (1-1) Extract the camera position from the camera pose of the FFHQ portrait dataset, and represent it in spherical coordinates as (θ,φ,r), where r is always equal to 2.7;

[0014] (1-2) Using the Gaussian kernel estimation method, calculate the pose density function ρ of the FFHQ portrait dataset with respect to angles θ and φ. FFHQ .

[0015] In step (2), the small-pose face image refers to the density ρ FFHQThe image where (θ,φ) is greater than or equal to 0.4.

[0016] Preferably, in step (2), the specific steps of collecting portrait images from the Flickr image website, extracting the camera pose of the portrait images, and filtering out small-pose face images to obtain the dataset LPFF of large-pose face images are as follows:

[0017] (2-1) Collect portrait images from the Flickr image website. First, use Dlib to obtain the facial feature points of each portrait.

[0018] (2-2) For faces that cannot be detected by the Dlib method, face alignment is used to obtain the facial feature points of each portrait;

[0019] (2-3) Input the facial feature points and portrait images obtained in steps (2-1) and (2-2) into the face reconstruction model to obtain the camera pose;

[0020] (2-4) Input the angles θ and φ in the camera pose into the density function ρ of FFHQ. FFHQ The density ρ is obtained. FFHQ (θ,φ), remove all images with a density greater than or equal to 0.4;

[0021] (2-5) Crop all portrait images based on facial feature points so that the face is in the center of the image.

[0022] Preferably, in step (3), the Gaussian kernel estimation method is used to calculate the camera attitude density function ρ of FFHQ+LPFF with respect to angles θ and φ. FFHQ+LPFF .

[0023] Preferably, in step (3), the function used to repeatedly calculate the image based on the density of the corresponding camera pose is specifically:

[0024]

[0025] Here, α is a parameter that controls sampling, and N represents the number of times the image is repeated.

[0026] Preferably, in step (4), the specific steps for training a geometrically more realistic 3D face-perceiving GAN model using the FFHQ+LPFF and FFHQ+LPFF-rebal datasets are as follows:

[0027] (4-1) Using the EG3D model structure, the network was retrained using the FFHQ+LPFF dataset to obtain the model.

[0028] (4-2) Using FFHQ+LPFF-rebal to train the The model was fine-tuned to obtain Model.

[0029] Preferably, in step (4-2), the step of using FFHQ+LPFF-rebal to process the trained data... The model is fine-tuned to use the trained model. After initializing the model's weight parameters, the model is trained again on a new dataset.

[0030] The present invention also provides a geometrically more realistic 3D perceptual face GAN model generation device, including a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor, wherein the computer processor executes the training method of the geometrically more realistic 3D perceptual face GAN model.

[0031] Compared with the prior art, the present invention has at least the following beneficial effects:

[0032] 1. This invention proposes a data collection process for extracting large-pose face images, establishes a high-quality large-pose face dataset, can better collect large-pose faces, and solves the problem of insufficient large-pose data in existing face datasets.

[0033] 2. This invention trains a more geometrically realistic 3D perceptual face GAN model, which can better handle large-pose faces and generate face geometry that conforms to the physiological structure of real faces. Using the training method of this invention to reconstruct faces in 2D images, high-quality multi-view rendering results and realistic 3D geometry can be obtained. Therefore, by using the method of this invention to predict 3D faces from 2D images, it is possible to reconstruct 3D faces or make a still real photo speak. Attached Figure Description

[0034] Figure 1 This is a flowchart illustrating the training method for a geometrically more realistic 3D perceptual face GAN model in this embodiment.

[0035] Figure 2 This refers to a portion of the large-pose face image dataset obtained in step (2) of the embodiment;

[0036] Figure 3 The result is the generation of a more geometrically realistic 3D perceptual face GAN model in the example. Detailed Implementation

[0037] like Figure 1 As shown, the training method for a geometrically more realistic 3D perceptual face GAN model includes the following steps:

[0038] (1) Using the camera poses of the FFHQ portrait dataset, calculate the camera pose density function ρ of the FFHQ portrait dataset with respect to angles θ and φ. FFHQ .

[0039] (1-1) Extract the camera position RT = (x, y, z) from the camera pose C = [R|RT] of all images in the FFHQ portrait dataset. Represent (x, y, z) in spherical coordinates as (θ, φ, r), where r is always equal to 2.7.

[0040] For camera pose data for the FFHQ portrait dataset, see “Efficient geometry-aware 3D generative adversarial networks”, The IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 16123-16133, 2022.

[0041] For methods on camera pose extraction, see “Accurate 3D face reconstruction with weakly-supervised learning: From single image to image set”, The IEEE / CVF conference on computer vision and pattern recognition workshops, 2019.

[0042] (1-2) Using the Gaussian kernel estimation method, calculate the pose density function ρ of the FFHQ portrait dataset with respect to angles θ and φ. FFHQ .

[0043] (2) Collect portrait images from the Flickr image website, extract the camera pose of the portrait images, and filter out small pose face images to obtain a dataset containing 19,950 large pose face images, denoted as LPFF. The dataset obtained by combining the FFHQ dataset and the LPFF dataset is denoted as FFHQ+LPFF.

[0044] (2-1) Collect 155,720 portrait images from the Flickr image website. First, use Dlib to obtain the facial feature points L of each portrait X;

[0045] For the Dlib face feature point extraction method, see "One millisecond face alignment with an ensemble of regression trees", The IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 1867–1874, 2014.

[0046] (2-2) For faces that cannot be detected by the Dlib method, face alignment is used to obtain the face feature points L of each portrait X;

[0047] For methods on face alignment and facial landmark extraction, see “How far are we from solving the 2D & 3D face alignment problem, and a dataset of 230,000 3D facial landmarks”, The IEEE International Conference on Computer Vision, ICCV, pp. 1021–1030, 2017.

[0048] (2-3) Input the facial feature points L and the image X into the face reconstruction model to obtain the camera pose C = [R|RT];

[0049] For methods on camera pose extraction, see “Accurate 3D face reconstruction with weakly-supervised learning: From single image to image set”, The IEEE / CVF conference on computer vision and pattern recognition workshops, 2019.

[0050] (2-4) Represent the camera position RT=(x,y,z) in the camera pose using spherical coordinates as (θ,φ,r), and input θ,φ into the density function ρ of FFHQ. FFHQ The density ρ is obtained. FFHQ (θ,φ), remove all images with a density greater than or equal to 0.4;

[0051] (2-5) All portrait images are cropped based on facial feature points so that the face is in the center of the image. The final 19,950 images with a density of less than 0.4 and the face in the center of the image are the large pose image dataset LPFF.

[0052] Figure 2 The image shown is a portion of the large-pose face image dataset obtained in steps (2-5). It can be seen that in the large-pose face image dataset LPFF of this invention, all images are high-resolution (1024x1024 resolution), without significant noise, and exhibit large poses, as well as good diversity in terms of gender, ethnicity, age, expression, and lighting conditions.

[0053] (3) Resample the FFHQ+LPFF dataset to make it more balanced. Calculate the camera pose density function ρ of FFHQ+LPFF with respect to angles θ and φ using the Gaussian kernel estimation method. FFHQ+LPFF For all images in FFHQ+LPFF, the function used to perform several repeated calculations on the images based on the density of the corresponding camera pose is as follows:

[0054]

[0055] Here, α is a parameter controlling sampling, and N represents the number of times the image is repeated. The resampled FFHQ+LPFF is represented as FFHQ+LPFF-rebal.

[0056] (4) Use the two datasets FFHQ+LPFF and FFHQ+LPFF-rebal to train a geometrically more realistic 3D face perception GAN model.

[0057] (4-1) Using the EG3D model structure, the network is first retrained using the FFHQ+LPFF dataset to obtain the model.

[0058] For the model structure and training method of EG3D, please refer to "Efficient geometry-aware3D generative adversarial networks", The IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 16123-16133, 2022.

[0059] (4-2) Using FFHQ+LPFF-rebal to train the Fine-tuning the model, that is, using the trained model After initializing the model's weight parameters, the model is trained again on a new dataset to obtain... Model.

[0060] Figure 3 This is the result of generating a more geometrically realistic 3D perceptual face GAN model in this embodiment. Figure 3 In the diagram, odd-numbered rows represent multi-view face images obtained using the training method of this invention, while even-numbered rows represent their corresponding face geometry. From... Figure 3 It can be seen that the facial images obtained by this invention are of high quality and realistic, possessing a certain degree of diversity, and the facial geometry is very close to the three-dimensional geometry of a real human face. Therefore, it can be demonstrated that if the training method of this invention is used to reconstruct faces in two-dimensional images, high-quality multi-view rendering results and realistic three-dimensional geometry can be obtained.

Claims

1. A training method for a geometrically more realistic 3D perceptual face GAN model, characterized in that, Comprise the following steps: (1) Calculate the camera pose density function p of the FFHQ portrait dataset about angles Q, f using the camera poses of the FFHQ portrait dataset FFHQ ; (2) Collect portrait images from the Flickr image website, extract the camera pose of the portrait image, filter out small-pose face images, obtain a large-pose face image dataset LPFF, and combine the FFHQ dataset with the LPFF dataset to obtain a dataset FFHQ+LPFF; (3) Resample the FFHQ+LPFF dataset to obtain a resampled dataset FFHQ+LPFF-rebal, and calculate the camera pose density function p of FFHQ+LPFF with respect to angles Q, f FFHQ+LPFF For all images in FFHQ+LPFF, the images are calculated several times according to the density of the corresponding camera pose; In step (3), the function used for repeated calculation of the image according to the density of the corresponding camera pose is specifically: Wherein, α is a control sampling parameter, and N represents the repetition number of the picture; (4) Training a geometrically more realistic three-dimensional perception face GAN model using FFHQ+LPFF and FFHQ+LPFF-rebal two datasets.

2. The method of Claim 1, wherein, In step (1), the camera pose of the FFHQ portrait dataset is used to calculate the camera pose density function p of the FFHQ portrait dataset with respect to angles θ, φ FFHQ The specific steps are as follows: (1-1) Extract the camera position in the camera pose of the FFHQ portrait dataset, expressed in spherical coordinates as (θ, φ, r), where r is equal to 2.7; (1-2) Using a Gaussian kernel estimation method, the pose density function p of the FFHQ portrait dataset with respect to angles Q, f is calculated FFHQ .

3. The method of Claim 1, wherein, In step (2), the small-pose face image with density p FFHQ (θ, φ) greater than or equal to 0.

4.

4. The method of Claim 3, wherein, In step (2), the specific steps of collecting portrait images from the Flickr image website, extracting the camera pose of the portrait image, filtering out small-pose face images, and obtaining a large-pose face image dataset LPFF are as follows: (2-1) Collect portrait images from the Flickr image website, and use Dlib to obtain the face feature points of each portrait; (2-2) For faces that cannot be detected by the Dlib method, use face alignment to obtain the face feature points of each portrait; (2-3) Input the face feature points obtained in steps (2-1) and (2-2) and the portrait image into a face reconstruction model to obtain the camera pose; (2-4) Input the θ, φ angles in the camera pose into the density function ρ of FFHQ FFHQ , to get the density ρ FFHQ (θ, φ), remove all images whose density is greater than or equal to 0.4; (2-5) Crop all portrait images according to the face feature points, so that the face is in the center of the image. 5.The method of Claim 1, wherein, In step (3), the camera pose density function p of FFHQ+LPFF with respect to angles Q, f is calculated using a Gaussian kernel estimation method FFHQ+LPFF . 6.The method of training a geometrically more realistic three-dimensional perceptual face GAN model according to claim 1, wherein, In step (4), the specific steps of training a geometrically more realistic three-dimensional perception face GAN model using FFHQ+LPFF and FFHQ+LPFF-rebal two datasets are as follows: (4-1) Using the model structure of EG3D, retrain the network with the FFHQ+LPFF dataset to obtain the model (4-2) Fine-tune the trained model with FFHQ+LPFF-rebal to get the model. model.​ 7. The method of Claim 6, wherein the geometrically more realistic three- dimensional perceptual face GAN model is trained by, In step (4-2), the fine-tuning of the trained FFHQ+LPFF-rebal model is initialized by using the weight parameters of the trained FFHQ+LPFF-rebal model. After the initialization, the model is further trained on the new dataset.​ 8. A more geometrically realistic three-dimensional perceptual face GAN model generation apparatus comprising a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor, the apparatus being characterized by: The computer processor executes the training method of the geometrically more realistic three-dimensional perception face GAN model according to any one of claims 1-7.

Images