A face frontalization method based on feature perception under dual-mode face transformation

Abstract

This invention relates to the field of face frontalization technology, and more particularly to a face frontalization method based on feature perception under dual-mode face transformation, comprising: S1, inputting a face image at any angle; S2, obtaining a 3D model through a 3DDFA model; S3, using the 3D model to obtain a frontal, damaged face from the face image at any angle; S4, inputting the frontal, damaged face into a GAN network to obtain a lossless frontal face. This invention is based on a dual-mode generative adversarial network model to address the problem of face frontalization with different characteristics under different poses; leveraging the ability of a 3D model to fit faces in any pose, a 2D image is fitted into the 3D model, and face transformation is achieved by rotating the 3D model to a certain pose angle; to preserve the detailed information of the frontal face view, a perceptual loss module fusing content loss and identity loss is added to the generative network to improve the realism of the generated frontal view.

Description

Technical Field

[0001] This invention relates to the field of face frontalization technology, and in particular to a face frontalization method based on feature perception under dual-mode face transformation. Background Technology

[0002] With the development of deep learning and the acquisition of large amounts of facial images, face recognition methods have not only made great progress but have also been able to handle increasingly complex situations during face recognition. In pose-invariant face recognition research, there are two main approaches: one is to extract and learn facial features from different poses, and the other is to first uniformly adjust the face to a frontal view before feature extraction. Both methods encompass face frontalization and feature learning. However, in the latest face recognition methods, the performance difference between frontal-frontal and frontal-profile data on the CFP face verification benchmark database exceeds 10%. RAR has implemented an unsupervised face frontalization model driven by a large amount of data, achieving good results in frontalization of faces at small angles. DR-GAN and TP-GAN have learned face generation models applicable to face recognition, completing the conversion from large-pose faces to frontal faces, achieving some success. However, the accuracy of DR-GAN in generating and recognizing frontal profiles on CFP is not very high, which is also the source of the difficulty in face frontalization.

[0003] Currently, face frontalization methods include 3D-based methods like 3DMM (Hassner), deep learning methods like Yim, and statistical methods. The classic 3DMM model represents the 3D face model using vectors of shape and texture, simulating more faces through linear combinations of vectors that constitute the average face. In 2015, Hassner et al. also used an average 3D model to achieve face frontalization. With the development of deep networks and the demand for large-angle face frontalization, 2D and 3D texture distortion models based on deep networks, such as FFGAN and FFWM, have become mainstream. Yim is a typical 2D face frontalization model that reconstructs a target pose face by inputting a 2D image and Remote Code into a DNN network. With the introduction of Generative Adversarial Networks (GANs) by Goodfellow, their application in image generation has become increasingly widespread, leading to the emergence of more face frontalization networks based on 2D GANs. DRGAN applies the GAN network structure while maintaining a similar framework to Yim. DRGAN inputs an image, noise, and pose encoding into its GAN network to generate corresponding identity representations and a face with the target pose. A typical dual-channel face frontal view synthesis network generates a frontal view of the input image based on two networks: local facial features and global contours. With the introduction and continuous development of 3DMM models, the BFM database is widely used, allowing each 3D face to be represented with the same number of point clouds or facets. Methods for frontalizing faces based on 3D applications have been proposed. HF-PIM introduces 3D face UV maps along with dense correspondence fields into a CNN-based wrapper to generate a frontal face. UV-GAN applies the 3DMM model to obtain UV maps corresponding to the 3D face model. Facial textures invisible due to angle are filled with noise in the maps. The final result is that after training the network to generate complete UV maps, they are mapped onto the 3D model to obtain views from various angles, including the frontal view. The latest RAR model utilizes 3D modeling, rendering models, and GANs to achieve arbitrary angle rotation and synthesis of facial details.

[0004] Perceptual loss functions were proposed as early as 2016 and applied to style transfer and super-resolution image reconstruction. They are increasingly combined with adversarial networks for image or video super-resolution tasks. Introducing perceptual loss to replace pixel loss, and treating pixel loss as an MSE regression problem, helps to correctly determine image similarities even when pixel position deviations occur due to image displacement. SRGAN, to achieve realistic super-resolution image reconstruction, proposed a perceptual loss function combining content loss and adversarial loss. It adds a content similarity function to the perceptual similarity function, improving the detail of super-resolution image reconstruction. ESRGA, as an enhanced SRGAN, improved the network structure while retaining perceptual loss as the core loss function for super-resolution image generation. Summary of the Invention

[0005] The technical problem to be solved by this invention is that: the frontal images generated by traditional models are prone to losing the identity features of the input image; and traditional models cannot generate features of occluded parts well.

[0006] The technical solution adopted in this invention is: a face frontalization method based on feature perception under dual-mode face transformation, comprising the following steps:

[0007] S1. Input a face image from any angle;

[0008] S2. Obtain the 3D model through the 3DDFA model;

[0009] A face shape model can be represented as a vector S∈R 3N N is the number of vertices, and the specific formula is as follows:

[0010]

[0011] in, It is an average shape, where V = [v1, v2, ..., v m ]∈R 3N×m These are the principal components obtained by PCA, where α is the shape parameter;

[0012] With the continuous development of 3DMM, the new 3D face learning model is:

[0013]

[0014] in, It is an average shape, where V = [v1, v2, ..., v m ]∈R 3N×m These are the principal components obtained from PCA, where α is the shape parameter, and e i The principal components corresponding to the shape are used to fit the 3D face model to the 2D image. Then, through orthogonal projection, the model is projected onto a two-dimensional plane, yielding the following formula:

[0015] P=f*Pr*R*S+t (3)

[0016] Where Pr is the orthogonal projection matrix, R is the rotation matrix, f is the scaling factor, t is the displacement vector, and the set of model parameters to be solved is parameters = [f, R, t, α, β].

[0017] S3. Use a 3D model to obtain a frontal, damaged face from a face image at any angle;

[0018] The 3D model is obtained from an existing 3DDFA model, using the following formula, where I is the original input image:

[0019] M(P)=Fitting(I) (4)

[0020] The rotation matrix of the corresponding image is R. Since texture also plays an important role in the face frontalization process, the texture corresponding to each vertex is solved by perpendicular (orthogonal) projection, specifically as follows:

[0021] T=I(V(p)) (5)

[0022] Where P is the position information of each fixed point, v is the function for calculating the texture, and I is the function for calculating the vertical projection;

[0023] S4. Input the damaged frontal face into the GAN network to obtain a lossless frontal face.

[0024] Overall network loss total Represented as:

[0025] l total =λ gan l GAN +l p (6)

[0026] Where, λ gan Let λ be the weighting parameter. gan =10, l GAN Represents the loss of generative adversarial networks, l p The loss represents the feature perception module loss;

[0027] l GAN The specific formula is as follows:

[0028]

[0029] Among them, I input For the original image, I GT Represents the ground truth of an image, G(I) input ) for generated fake images, p a (I GT ) represents the true data distribution, p b (I input ) represents the distribution of fabricated data;

[0030] The core idea of ​​formula (7) is to iteratively train and generate a discrimination network; input image I input An attempt is made to train a generative network G capable of deceiving a discriminative network D; the discriminative network is trained to distinguish between real images and fake images generated by the generative network; through l GAN The loss function is the function that we want the parameters to follow a p-value. a (I GT (When the real data distribution) and the parameters follow p b (Iinput (The distribution of falsified data) shows an opposite trend;

[0031] Loss function of feature perception module l p The formula is as follows:

[0032]

[0033] Where, λ cte and λ vgg These represent the corresponding weight parameters, which in this paper are 1 and 10, respectively. These represent the loss values ​​of the input image when using the CTE network with the MAE and MSE loss functions, respectively. The values ​​represent the loss values ​​of the input image when using the VGG network with the MAE and MSE loss functions, respectively.

[0034] The beneficial effects of this invention are:

[0035] 1. A dual-mode generative adversarial network model is used to address the problem of frontalizing faces with different characteristics under different poses. Taking advantage of the ability of 3D models to fit faces in any pose, a 2D image is fitted into a 3D model, and the face is transformed by rotating the 3D model to a certain pose angle. Data pairs based on two training modes based on face transformation are designed for training. In order to preserve the detailed information of the frontal face view, a perceptual loss module that integrates content loss and identity loss is added to the generative network to improve the realism of the generated frontal view. Attached Figure Description

[0036] Figure 1 This is a model framework diagram of the present invention;

[0037] Figure 2 This is a schematic diagram of the attitude transformation in two modes of the present invention;

[0038] Figure 3 This is a network structure diagram of the present invention;

[0039] Figure 4 This is an iterative graph of the MAE and MSE loss functions of the present invention. Detailed Implementation

[0040] The present invention will be further described below with reference to the accompanying drawings and embodiments. The drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0041] This invention proposes a feature-aware face frontalization method under dual-mode face transformation, which can address the problem of frontalizing faces with different features under different poses. To handle wild human face datasets, a 3D face model is applied; faces in different poses are fitted into the 3D model, rotated, and rendered onto a 2D plane, forming image pairs with Groundtruth values ​​for network training. A feature-aware module is introduced, enhancing the generation of detailed features through training with two loss functions. Images generated by the feature-aware model are visually more attractive and easier to apply to face recognition.

[0042] like Figure 1 The present invention provides a model framework diagram. A method for frontalizing a face under dual-mode face transformation based on feature perception includes the following steps:

[0043] S1. Input a face image from any angle;

[0044] S2. Obtain the 3D model through the 3DDFA model;

[0045] The classic 3D deformable model (3DMM) proposes a statistical model of face shape and appearance. It uses PCA to reduce the dimensionality of 3D face data obtained from instrument scanning, and then combines and deforms the reduced data to obtain new 3D face data, i.e., the newly generated face, thus ensuring the model's variability, as shown in Equation 1. With the continuous development of 3DMM, the difficulty of learning the texture part of the 3D model in the classic 3DMM has led to its replacement by textures directly extracted from images, which are no longer considered for solving. Another new element affecting face shape learning, facial expression, has also become a target to be learned. Consequently, the 3DMM expression model, which incorporates expression learning, has emerged, with the BFM model being the most widely used, as shown in Equation 2.

[0046] S3. Use a 3D model to obtain a frontal, damaged face from a face image at any angle;

[0047] The first method is shown in Formula 4. The frontal face with loss is I0, the 3D face model is M(P), and the corresponding pose is R. The face pose of the model is transformed R->R0 to obtain a new 3D face model M(P0). R0 is the rotation matrix of the corresponding frontal face model, which is obtained by multiplying R by the rotation matrix. The input image texture T is obtained, and the texture T and the rotated 3D frontal face model are rendered into a 2D image to obtain the frontal face with loss I0.

[0048] The second type is a frontal lossy face I′, with a 3D face model M(P) generated from the input image, corresponding to pose R. The model's facial pose is transformed, R→R. m A new 3D face model M(P) was obtained. m ), R mThe rotation matrix corresponding to the target pose face model is obtained by multiplying R by the rotation matrix. The texture T of the input image I is then obtained. The texture and the rotated 3D frontal face model are used to render a 2D lossy face I. m Second image acquisition I m Texture T m , by texture T m The 3D frontal face model M(P) of image I is rendered into a 2D image to obtain a frontal face with loss I′;

[0049] The network input sets for the two schemes are I. input = {I0, I′}, where I is the ground truth set corresponding to the two schemes. GT ={I F ,I},I F This represents the frontal true image corresponding to input image I; the specific pose transformation is as follows: Figure 2 As shown.

[0050] S4. Input the damaged frontal face into the GAN network to obtain a lossless frontal face.

[0051] Specifically, the network includes a generator network G and a discriminator network D, applying generative adversarial networks to face frontalization; Equation 7 solves the max-min problem by alternately training the generator network and the discriminator network.

[0052] The core idea of ​​Formula 7 is to iteratively train and generate a discriminative network; I input The input is fed into a generator network in an attempt to train the generator to produce an image G(I) that can fool the discriminator network. input The discriminant network trains a discriminator to distinguish real images and generates fake images G(I) generated by the generator network. input The generator network consists of nine ResNet blocks, each containing two convolutional layers with ReLU activation. Each convolutional layer has a 3×3 filter kernel and 1024 feature maps. The discriminator network comprises five convolutional layers with varying numbers of 4×4 filter kernels. To ensure the discriminator network can correctly distinguish between real and fake images, we appropriately increased the number of convolutions and compared image features at different scales. Both the generator and discriminator networks underwent continuous training to solve the minimization and maximization problems in the equation. Figure 3 A detailed network structure diagram is shown.

[0053] Since feature perception is crucial for the performance of reconstructing frontal faces, a pre-trained VGG network with good identity preservation capabilities is adopted. The entire perception module is represented by two modules: content loss and identity loss, as shown in Equation 8. During training, in addition to ensuring the generation of complete facial images, the network generates a reconstructed image with the same ID (identity representation) as the input image. The identity loss applies the pre-trained VGG model. The ground truth and the original image are processed through the output of the five-layer network, and a weighted combination of the L loss is calculated. The loss L includes the following three cases: L = {L1, L2, SmoothL1}. The VGG loss formula is as follows:

[0054]

[0055] Where, σ i I represents the weights corresponding to the output of each layer of the network. GT Representing the ground truth and I of the image input This represents the original image, where W and H represent the image's width and height, respectively.

[0056] The SmoothL1 loss is defined as follows:

[0057]

[0058] Where x is the value in formula (9) The value;

[0059] Replace L1 or L2 loss with SmoothL1 because SmoothL1 overcomes the shortcomings of the former, converging to higher accuracy while being less susceptible to noise (robust).

[0060] The pixel-guided MSE content loss estimate is:

[0061]

[0062] in, G(I) represents the value of each pixel in the Ground Truth. input ) x,y The input image is processed by the generator network, where W and H are the width and height of the image, respectively.

[0063] During the initial model training process, the MSE loss value is relatively large, but as the training progresses, the overall loss value shows convergence.

[0064] Trained with adversarial network loss, the network can generate faces with different IDs. Adding perceptual loss during training ensures consistency in the identities of the generated faces. As content loss is incorporated into network training, the pixel count of the generated faces increases. The generative adversarial network loss can be represented as l. GAN The total loss is expressed as in Formula 8; the changes in MAE and MSE losses are as follows: Figure 4 As shown.

[0065] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A method for face frontalization under dual-mode face transformation based on feature-awareness, characterized in that, The method comprises the following steps: S1, inputting a face image of an arbitrary angle; S2, obtaining a 3D model through a 3DDFA model; S3, obtaining a front face with loss from the face image of the arbitrary angle according to the 3D model; positive face loss face is a positive face loss face with the positive face loss face input set is the corresponding ground truth set is , denotes a positive real image of the image ​ Frontal loss face Comprising: corresponding pose of 3D face model M (P) is R, rotation model face pose R->R0, obtain new 3D face model M (P0) ;R0 is the pose after corresponding frontal face model is rotated, R0 is obtained by R multiplied by first rotation matrix;Obtain image Texture T, by texture T and the 3D face model after rotation are rendered to 2D image, obtain Frontal loss face of the frontal loss face; There is a loss of face on the front This includes: converting the pose of the 3D face model M(P) to R, and transforming the model's facial pose. A new 3D face model was obtained. , To correspond to the pose of the target pose face model, by Multiply by the second rotation matrix to obtain Acquire image Texture T, rendered from texture T and a rotated 3D face model into a 2D frontal lossy face. ; Acquire image Texture after the third rotation matrix , by texture and images 3D face model Render to a 2D image to obtain The front view shows a loss of facial features; S4, inputting the front face with loss into a GAN network to obtain a front face without loss.

2. The method of claim 1, wherein, The formula for obtaining the 3D model through the 3DDFA model is: (2) where is the principal component corresponding to the shape related coefficient; n represents the number of principal components obtained by another group of PCA for describing the change of face shape; the face shape model is represented as a vector , is the number of vertices, is the average shape, wherein is the principal component obtained by PCA, is the shape parameter, is the principal component corresponding to the shape, fitting the 3D face model to the 2D image, projecting onto the two-dimensional plane by orthogonal projection, and the following formula is obtained: (3) wherein, is an orthogonal projection matrix, is a rotation matrix, is a scale factor, is a displacement vector, and the set of solved model parameters is parameters .

3. The method of claim 1, wherein: The inputting of the damaged front face into the GAN network to obtain the undamaged front face is to apply the generative adversarial network to the front face work, and the generative adversarial network loss is represented as , The formula is as follows: （7） wherein, is the original image, is the Ground truth representing the picture, is the generated fake image, is the real data distribution, is the fake data distribution; The pre-training network VGG with good identity preservation ability is adopted, the whole perception module is represented as two modules of content loss and identity loss, and the feature perception module loss function The formula is as follows: （8） Wherein, Loss values of MAE, MSE loss function respectively used when the input image uses the CTE network, Loss values of MAE, MSE loss function respectively used when the input image uses the VGG network, And Respectively represent the weight parameters of And Respectively represent the weight parameters of In the training process, in addition to ensuring that a complete face image is generated, the network generates a reconstruction image with the same id as the input picture, the identity loss applies a pre-trained VGG model, the ground truth of the picture and the original image pass through the output of five layers of network, and the weighted combination of L loss is calculated.

4. The method of claim 3, wherein: The L loss includes three cases: ; The VGG loss is defined as follows: （9） wherein, represents the weight corresponding to the output of each layer of the network, represents the weight corresponding to the output of each layer of the network, represents the Ground truth of the picture and represents the original picture, W and H respectively represent the width and height of the picture: The SmoothL1 loss is defined as follows: （10） where x is the value of formula (9) ; The L1 or L2 loss is changed to SmoothL1, and the pixel-guided MSE content loss is estimated as follows: （11） wherein, is the value of each pixel point in Groundtruth, is the value of each pixel point in the input picture after passing through the generation network, W and H are the width and height of the picture, respectively.

5. The method of claim 3, wherein: The discrimination network is a discriminator that discriminates between images input to the generation network The input to the generation network attempts to train the generator to generate images that fool the discriminator ; The discriminator network trains the discriminator to distinguish between real images and fake images generated by the generator network The generator network includes nine ResNet blocks in the middle, each of which includes two convolutional layers with ReLU as the activation function, each convolutional layer having a 3x3 filter kernel and 1024 feature maps; the discriminator network includes five convolutional layers with 4x4 filters.

Images