Facial texture diversity completion method for 3D face model based on PIC

Abstract

A method for completing facial texture diversity of a 3D face model based on PIL, a high-definition facial texture dataset is constructed through a texture completion network and a facial texture dataset; then a convolutional variational autoencoder (CVAE) is used to estimate the parameter distribution of the latent space of the facial texture mask area, and the implicit information in the parameter distribution is decomposed after sampling, and then the distribution regularization loss and the appearance matching loss are used as the generation loss to train the improved PIC network model, and finally the trained improved PIC network model is used to generate a complete facial texture to replace the 3D face model with incomplete texture. The present invention utilizes texture restoration and facial images of different angles to generate a complete facial texture for a 3D face model reconstructed from a single image.

Description

Technical Field

[0001] The present invention relates to a technology in the field of image processing, specifically a method for completing the facial texture diversity of a 3D face model based on Pluralistic Image Completion (PIC). Background Art

[0002] Existing deep learning-based 3D face reconstruction algorithms can quickly generate 3D model results of input images, but they still face many problems during the training process. One of the most critical issues is the clarity and completeness of the texture of the reconstructed face. An obvious problem with single-image 3D face reconstruction is that a single image cannot show the complete face texture, especially side-view images. How to restore the missing side texture as much as possible is a very important issue.

[0003] The face parameterized models designed after the deformation model of 3D face synthesis also contain texture parameters. Therefore, given a person's picture, the facial texture information closest to the person can be obtained by fitting the texture parameters. However, this method has high requirements on the face database used by the parameterized model. For example, the facial texture fitting effect of the parameterized model using the Caucasian face database is relatively poor for Asians. Secondly, the facial texture fitting for non-skin images is seriously distorted, such as images of ancient people. UV-GAN [Deng J, Cheng S, Xue N, et al. UV-GAN: Adversarial Facial UV Map Completion for Pose-invariant Face Recognition [J]. 2017.] uses the powerful ability of generative adversarial networks to convert facial textures into images - image problems. Use the real pixels in the image to construct the facial texture closest to the given image. LAB[Wu W, Qian C, Yang S, et al. Lookat Boundary: A Boundary-Aware Face Alignment Algorithm[C] / / 2018IEEE / CVFConference on Computer Vision and Pattern Recognition.IEEE, 2018.] uses the facial texture map constructed by PRNet[Zhu X, Zhen L, Liu X, et al. Face Alignment Across Large Poses: A 3D Solution[C] / / 2016IEEE Conference on Computer Vision and Pattern Recognition(CVPR).IEEE, 2016.] to design Attention ResCUNet-GAN to directly generate complete facial textures. This saves the time of UV-GAN to obtain incomplete facial textures. UV-GAN and LAB are mainly used to solve the occlusion caused by the head posture of the person in the input image, and do not pay attention to the situation where the face is occluded by other objects.IS Na et al. [Na IS, Tran C, Nguyen D, et al. Facial UV map completion for pose-invariant face recognition: a novel adversarial approach based on coupled attention residual UNets [J]. Human-centric Computing and Information Sciences, 2020, 10(1): 1-17.] designed a generative adversarial network specifically for this problem, which can input images of people with facial occlusions and output complete facial images. Some of the training sets are based on random masks added to the facial key points of the image, such as sunglasses and glasses masks on the eyes, masks on the mouth, etc. Summary of the invention

[0004] In view of the defects that the texture clarity and completeness of the face reconstructed by the prior art are low, the single-image 3D face reconstruction cannot display the complete facial texture, and the facial texture details are generally restored through interpolation processing combined with the texture map, while if the single-view face provides a side image, the occluded facial details cannot be restored through interpolation processing. A PIL-based facial texture diversity completion method for a 3D face model is proposed. By using texture restoration and face pictures at different angles, a complete facial texture can be generated for the 3D face model reconstructed by a single image.

[0005] The present invention is achieved through the following technical solutions:

[0006] The present invention relates to a facial texture diversity completion method for a 3D face model based on PIC. A high-definition facial texture dataset is constructed by using a texture completion network and a facial texture dataset. Then, a convolutional variational autoencoder (CVAE) is used to estimate the parameter distribution of a latent space of a facial texture mask region. After sampling, the implicit information in the parameter distribution is decomposed. Then, a distribution regularization loss and an appearance matching loss are used as generation losses to train an improved PIC network model. Finally, the trained improved PIC network model is used to generate a complete facial texture to replace a 3D face model with an incomplete texture.

[0007] The present invention relates to a system for implementing the above method, comprising: an incomplete texture generation unit, a residual encoder, a residual decoder and a long-short attention unit, wherein: the incomplete texture generation unit generates an incomplete texture image according to a 3D face reconstruction model and a public face database; the residual encoder encodes the incomplete texture image to obtain implicit information, and samples the implicit information; the residual decoder decodes the implicit information to obtain a complete texture image; the long-short attention unit finally obtains a variety of facial texture completion based on the encoded information of the incomplete texture image and the sampled implicit information.

[0008] Technical Effects

[0009] The present invention creates an algorithm for constructing a facial texture dataset and improves the face completion PIC framework to apply it to facial texture completion. A large number of high-definition side-view images of people are generated by modifying the semantic vector of Seeprettyface StyleGAN, and incomplete facial textures are generated through a 3D face reconstruction framework such as 3DDFA / PRNet, and a high-definition facial texture dataset is obtained by symmetrical completion. The present invention is based on improving PIC and applying it to facial texture completion, eliminating the reconstruction path of PIC, and changing the local appearance matching loss to the global appearance matching loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a flow chart of the present invention;

[0011] Figure 2 Schematic diagram of the network architecture of the embodiment. DETAILED DESCRIPTION

[0012] like Figure 1 As shown, this embodiment relates to a method for completing facial texture diversity of a 3D face model based on PIC, comprising:

[0013] Step 1: Build a high-definition facial texture dataset, including:

[0014] a1. The high-definition facial texture dataset uses the public high-definition face generation preprocessing model of SeePrettyFace based on StyleGAN to establish a mapping relationship from (18,512)-dimensional vectors to (1024,1024,3)-dimensional vectors; then, in the pictures randomly generated by the high-definition face generation preprocessing model, a side view of a person can be found. By fixing its corresponding semantic vector in the (18,512)-dimensional vector (0:18,200:400), and further randomly generating the remaining positions, a large number of high-definition side view photos of people can be quickly obtained.

[0015] The high-definition face generation preprocessing model uses a (18,512)-dimensional vector to represent each face, and a slightly different new face appearance can be generated by just making a slight change to the vector.

[0016] The mapping relationship refers to: the high-definition face generation preprocessing model establishes a mapping relationship from a (18,512)-dimensional vector to a (1024,1024,3)-dimensional vector, and the latter has some attributes, such as age, gender, expression, etc.

[0017] a2. Generate the corresponding high-definition incomplete texture according to the high-definition face profile dataset obtained in step a1 using the pre-trained Mobilenet-based network model, and obtain the high-definition complete texture dataset through mirror symmetry processing;

[0018] The architecture of the Mobilenet-based network model is implemented using, but not limited to, 3DDFA ([Zhu X, Zhen L, Liu X, et al. Face Alignment Across Large Poses: A 3D Solution [C] / / 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2016.])

[0019] a3. Add false lighting and false shadows to each complete texture in the high-definition complete texture dataset, and finally obtain a pair of incomplete lighting face texture and complete lighting face texture data.

[0020] Step 2: Obtain the potential vector z through the encoding network E c , specifically including:

[0021] b1. Using random texture mask to obtain incomplete texture I m And the corresponding complete texture I g 、Missing part I c ;

[0022] The encoding network E is implemented using the PIC-based encoding network E ([Zheng C, Cham TJ, JCai. Pluralistic Image Completion[C] / / 2019IEEE / CVF Conference on Computer Vision and Pattern Recognition(CVPR).IEEE, 2019.]).

[0023] The random texture mask refers to: a random strip / block mask and a facial mask for facial images, that is, a front / left side / right side facial texture mask.

[0024] b2. Use conditional variational autoencoder CVAE to obtain the parameter distribution of the latent space, and then sample the latent vector z from the parameter distribution of the latent space c , z c Contains information about the missing region, specifically: Calculate the variational lower limit of the conditional log-likelihood function logp(Ic|Im) of the training instance: logp(Ic|Im)≥-KL(q ψ (z c |I c )||p φ (z c |I m ))+E qψ (z c |I c )[logpθ(I c |z C , I m )], where: I g is the original image, I m is the observable part, I c is the missing part. The KL divergence term represents the important sampling function q that it will learn. ψ (·|I c ) is regularized to a fixed latent prior

[0025] Step 3: Obtain the implicit information z encoded by the encoding network E through the decoder c Get information about missing areas;

[0026] The missing area refers to: due to the missing part of the image I c Used to infer the important function q during training ψ (·|I c )=N ψ (·). Therefore, the sampled latent vector z c Contains information about missing regions.

[0027] Step 4: Obtain different generation results by generating the network G and the implicit information z obtained in step 2. Specifically, when the important function q ψ (·|I c ) when sampling, the information containing the missing area can be obtained, and the likelihood function p θ (I c |z C , I m ) Focus on I c On the contrary, from the learning does not contain Ic The conditional prior p φ (·|I m ) When sampling, let the likelihood function model be related to I c The original instance is irrelevant and can promote creative generation.

[0028] The generative network G is implemented using a PIC-based generative network G (PIC[Zheng C, Cham TJ, JCai.Pluralistic Image Completion[C] / / 2019IEEE / CVF Conference on Computer Vision and Pattern Recognition(CVPR).IEEE, 2019.]).

[0029] Step 5: Minimize the adversarial loss, update the encoding network parameters and the generating network parameters through back propagation until the loss converges, including: Minimize the adversarial loss Among them I g沈n To generate an image, I g is a real image, and the discriminant network parameters θ are updated by back propagation D , minimize the distribution regularization loss Among them: the conditional priors of learning It is also a Gaussian distribution, regularized to q ψ (·|I c ) and minimize the appearance matching loss Where: I g沈n To generate an image, I g For real images.

[0030] like Figure 2 As shown, the system for implementing the above method involved in this embodiment includes: an incomplete texture generation unit, a residual encoder, a residual decoder and a long-short attention unit, wherein: the incomplete texture generation unit generates an incomplete texture image according to a 3D face reconstruction model and a public face database; the residual encoder encodes the incomplete texture image to obtain implicit information, and samples the implicit information, the residual decoder decodes the implicit information to obtain a complete texture image, and the long-short attention unit obtains a variety of facial texture completion based on the encoded information of the incomplete texture image and the sampled implicit information.

[0031] The incomplete texture generation unit includes: a 3D face model generation unit, an interpolation facial texture generation unit, and an incomplete facial texture generation unit, wherein: the 3D face model generation unit constructs a 3D face model according to the input facial image information, the interpolation facial texture generation unit performs image interpolation processing according to the vertex information and facial image information of the 3D face model to obtain the interpolated facial texture result, and the incomplete facial texture generation unit obtains the facial texture of the visible part according to a layer of depth information of the intercepted surface to obtain the incomplete facial texture result.

[0032] After specific practical experiments, under the specific environment settings of Tesla P40 GPU and Python PyTorch framework, with λ=10 -4 We train from scratch with a fixed learning rate of , using Adam optimization, β1 = 0 and β2 = 0.999. The final weight used is α KL =α app =20,α ad = 1. The above method is run with the parameters. The training time of the model with random irregularity and no central hole is about 5 weeks. The average inference time of the project is 59ms.

[0033] Compared with the existing technology, this method can construct a facial texture dataset at zero cost. By improving PIC and applying it to facial texture completion, eliminating the reconstruction path of PIC and changing the local appearance matching loss to the global appearance matching loss, the texture completion can be more complete and the completed texture is richer and clearer. At the same time, this is the first existing facial texture diversity completion method for 3D face models.

[0034] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A method for completing facial texture diversity of a 3D face model based on PIC, characterized in that: A large number of high-definition side-view images are generated by modifying the semantic vector of Seeprettyface StyleGAN, and incomplete facial textures are generated through a 3D face reconstruction framework, and a high-definition facial texture dataset is obtained by symmetrical completion; then a convolutional variational autoencoder is used to estimate the parameter distribution of the latent space of the facial texture mask area, and the decoder is used to obtain the implicit information encoded by the encoding network to obtain the information of the missing area, and then different generation results are obtained through the generation network and implicit information, minimizing the adversarial loss, minimizing the distribution regularization loss, and minimizing the appearance matching loss. The encoding network parameters and the generation network parameters are updated through back propagation until the loss converges. Finally, the trained improved PIC network model is used to generate a complete facial texture to replace the 3D face model with incomplete texture; The high-definition facial texture dataset is constructed in the following way, specifically including: a1. The high-definition facial texture dataset uses a high-definition face generation preprocessing model based on StyleGAN to establish a mapping relationship from a (18,512)-dimensional vector to a (1024,1024,3)-dimensional vector; then find a side view of a person in the randomly generated pictures by the high-definition face generation preprocessing model, fix its corresponding semantic vector in the (18,512)-dimensional vector (0:18,200:400), and further randomly generate the remaining positions, so as to quickly obtain a large number of high-definition side view photos of people; a2. Generate the corresponding high-definition incomplete texture according to the high-definition face profile dataset obtained in step a1 using the pre-trained Mobilenet-based network model, and obtain the high-definition complete texture dataset through mirror symmetry processing; a3. Add false lighting and false shadows to each complete texture in the high-definition complete texture dataset, and finally obtain the incomplete lighting face texture and complete lighting face texture data pairs; The improved PIC network refers to: removing the reconstruction path of PIC and changing the local appearance matching loss to the global appearance matching loss; The aforementioned minimization of adversarial loss, minimization of distribution regularization loss, and minimization of appearance matching loss, updates the encoding network parameters and the generating network parameters through back propagation until the loss converges, minimizing the adversarial loss Where: I gen To generate an image, I g is a real image, and the discriminant network parameters θ are updated by back propagation D ; Minimize the distribution regularization loss Among them: the conditional priors of learning It is also a Gaussian distribution, regularized to q ψ (·|I c ) and minimize the appearance matching loss Where: I gen To generate an image, I g is the real image, I m is the observable part, I c For the missing part.

2. The method for completing facial texture diversity of a 3D face model based on PIC according to claim 1, characterized in that: The high-definition face generation preprocessing model uses a (18,512)-dimensional vector to represent each face, and a slightly different new face appearance can be generated by just making a slight change to the vector.

3. The method for completing facial texture diversity of a 3D face model based on PIC according to claim 1, characterized in that: The parameter distribution of the latent space of the facial texture mask area is obtained by using random texture mask to obtain the observable incomplete texture part I m And the corresponding original image I g 、Missing part I c ; Then the conditional variational autoencoder CVAE is used to obtain the parameter distribution of the latent space, and then the latent vector z is sampled from the parameter distribution of the latent space c , z c Contains the information of the missing area; then the decoder is used to obtain the implicit information z encoded by the encoding network E c Get information about missing regions.

4. The method for completing facial texture diversity of a 3D face model based on PIC according to claim 3, characterized in that: The random texture mask refers to: a random strip / block mask and a facial mask for facial images, that is, a front / left side / right side facial texture mask.

5. The method for completing facial texture diversity of a 3D face model based on PIC according to claim 3, characterized in that: The sampling refers to: calculating the conditional log-likelihood function logp(I c |I m )’s variational lower bound: Among them: The KL divergence term represents the important sampling function q that it will learn ψ (·|I c ) is regularized to a fixed latent prior 6. A system for implementing the method for completing facial texture diversity of a 3D face model based on PIC as described in any one of claims 1 to 5, characterized in that: include: An incomplete texture generation unit, a residual encoder, a residual decoder, and a long-short attention unit, wherein: the incomplete texture generation unit generates an incomplete texture image according to a 3D face reconstruction model and a public face database; the residual encoder encodes the incomplete texture image to obtain implicit information, and samples the implicit information; the residual decoder decodes the implicit information to obtain a complete texture image; the long-short attention unit combines the encoded information of the incomplete texture image with the sampled implicit information to finally obtain a variety of facial texture completion; The incomplete texture generation unit includes: a 3D face model generation unit, an interpolation facial texture generation unit, and an incomplete facial texture generation unit, wherein: the 3D face model generation unit constructs a 3D face model according to the input facial image information, the interpolation facial texture generation unit performs image interpolation processing according to the vertex information and facial image information of the 3D face model to obtain the interpolated facial texture result, and the incomplete facial texture generation unit obtains the facial texture of the visible part according to a layer of depth information of the intercepted surface to obtain the incomplete facial texture result.

Images