A controllable face generation method based on facial micro-expressions

Abstract

The present invention discloses a controllable face generation method based on facial micro-expressions, comprising the following steps: (1) preprocessing of facial images; (2) constructing a convolutional neural network driven by facial micro-expressions; (3) generating controllable faces based on facial micro-expression parameters. By constructing a convolutional neural network, the present invention can extract the intrinsic movement details of a person's face using facial micro-expressions, so that the constructed network can artificially control the generation results while maintaining excellent image quality.

Description

Technical Field

[0001] The present invention relates to a controllable human face generation technology based on facial micro-expressions, and belongs to the technical field of digital image processing and deep learning. Background Art

[0002] There are usually two methods for face image generation technology: 1. Generative neural network based on real images; 2. Generative network based on parameter control, but both methods have certain defects.

[0003] 1. Generative neural network based on real images

[0004] Generative neural networks based on real images extract facial movements of people from input images and reconstruct the movements in generated images. On the one hand, researchers have collected large-scale multimodal datasets such as VoxCeleb and GRID to support the research of this task; on the other hand, researchers have conducted in-depth research on a variety of new face generation methods. This type of research can be divided into the following two aspects:

[0005] (1) Key point based methods. Compared with traditional methods, the most important thing about key point based methods is that they introduce pixel coordinate information. In image processing, key points provide detailed descriptions of image shapes and edges, which are more suitable for image generation tasks. Researchers have conducted in-depth research on key point based methods, which can generate accurate motion while retaining the identity of the person. For example, the unsupervised key point based FOMM method proposed by Siarohin et al. uses the affine transformation of the local area of ​​the key point to reconstruct the target motion using the spatial transformation method.

[0006] (2) Latent vector-based methods convert input information into latent vectors and use latent codes to represent character features. Unlike key point-based methods, latent code-based methods do not perform spatial transformations on images, but instead input latent vectors into convolutional neural networks to obtain results. Recent researchers have continuously proposed improvements to latent vector-based methods and have also obtained realistic results. For example, the PIRenderer method based on a single-dimensional latent vector proposed by Ren et al. uses adaptive instance normalization (AdaIN) to inject latent codes containing target information into convolutional neural networks, and finally uses adversarial generative networks (GANs) to train and learn the entire network.

[0007] Although generative neural networks based on real images have achieved relatively good generation results, this type of method can only process and reproduce expressions that exist in real images, without considering the controllability and interpretability of parameters, making it impossible to artificially control the generation details of the generation results of this type of method.

[0008] 2. Generative Neural Networks Based on Parameter Control

[0009] Generative neural networks based on parameter control refer to the artificial manipulation and attribute editing of static portrait images, such as changing the facial expressions of people. Researchers have conducted in-depth research on a variety of new facial editing methods. This type of research can be divided into the following two aspects:

[0010] (1) Macro-facial expression-based methods: Macro-facial expression-based methods are based on discrete emotion categories, such as happiness, surprise, fear, sadness, disgust, and anger. In facial attribute editing, macro-facial expressions provide a holistic description of facial movements, which affects the interpretability of the generated results. However, such methods cannot generate smooth facial expression transitions, which makes the generated results of face images not achieve the ideal effect.

[0011] (2) Methods based on facial micro-expressions. Most of the methods based on facial micro-expressions are based on the facial expression coding system proposed by Ekman and Friesen. This system defines facial micro-expressions as the combined result of coordinated movements of facial muscles, and uses action units (AU) to describe facial expressions. These action units are based on the contraction of specific facial muscles in biology and anatomy. Recently, researchers have also conducted research on methods based on facial micro-expressions. For example, the CycleGan-based Ganimation method proposed by Pumarola et al. uses the facial expression coding system to manipulate character expressions, and keeps the character's identity as accurate as possible by re-inputting the generated image into the encoder for training. Compared with methods based on macroscopic facial expressions, this method has more interpretable parameters and can artificially control the generation details.

[0012] Although the parameter-controlled generative neural network has achieved certain generation results, compared with other face generation methods, such as the generative neural network driven by real images, the existing methods still cannot generate sufficiently clear results. The present invention can solve this problem. Summary of the invention

[0013] In view of the problems and shortcomings in the prior art, in order to combine and retain the correlation between facial micro-expressions, the image generation quality of the convolutional neural network is further improved. The present invention provides a controllable face generation method based on facial micro-expressions. The network can not only obtain the same good generation effect, but also, compared with the existing method, the present invention can produce a specified generation result by artificially controlling the parameters, and has stronger controllability.

[0014] To achieve the above purpose, the technical solution adopted by the present invention is to convert facial micro-expression F∈R 17 Transformed into latent code V∈R M, where M is the potential dimension obtained by inputting the convolutional feature encoder, and then the face image S∈R W×H The convolution image generator with the potential code V as input generates a specified image, which specifically includes the following steps:

[0015] (1) Preprocessing of face images;

[0016] (2) Construct a convolutional neural network driven by facial micro-expressions;

[0017] (3) Controllable face generation based on facial micro-expression parameters;

[0018] The size of each face image is W×H. The datasets usually processed are between thousands and tens of thousands of images, including hundreds to thousands of different identities. The value range of W×H is between 128*128 and 256*256, and the parameters are subject to conditions such as the resolution when the image is collected.

[0019] Furthermore, the preprocessing of the face image in step (1) specifically includes the following steps:

[0020] (1-1) Use face-alignment to frame and crop the face area in the dataset, and remove face images that do not contain complete portrait faces and have blurred facial expressions.

[0021] (1-2) Use OpenFace to extract facial micro-expression features F∈R from the face region cropped in (1-1) 17 , and establish an index according to the data set storage format. F is a vector value based on the facial expression coding system, ranging from 0 to 1.

[0022] (1-3) The images in the dataset are divided into two parts: training data and test data in an 8:2 ratio according to actual needs. D 1 face images as training data, and the remaining (DD 1 ) face images as test data, where D represents the total number of face images in the dataset, and 0 <D1<D。

[0023] Furthermore, the step (2) of constructing a convolutional neural network driven by facial micro-expressions specifically comprises the following steps:

[0024] (2-1) Let facial micro-expression F∈R 17 Input to the convolutional feature encoder to obtain the potential vector V∈R M . Where M is the potential dimension obtained by inputting into the convolutional neural network feature encoder. The convolutional feature encoder here contains one-dimensional convolution and residual connection (skip-connect).

[0025] (2-2) Substitute the latent code V obtained in (2-1) and the face image S∈R W×H Input to the convolutional motion generator, output feature matrix Q∈R W×H . W×H is the size of the face image and feature matrix. The convolutional motion generator here includes the stacked hourglass network (stacked Hourglass Model) and adaptive instance normalization (AdaIN).

[0026] (2-3) Substitute the characteristic matrix Q∈R obtained from (2-2) W×H and face image S∈R W×H Input to the convolution image generator to get the output image The convolutional image generator here contains a generative adversarial network (GAN)

[0027] Furthermore, the method for generating a controllable face based on facial micro-expression parameters described in step (3) is: inputting the face image and the specified facial micro-expression into the convolutional neural network constructed in step (2) for generation, and obtaining the final generation result.

[0028] Beneficial effects:

[0029] (1) The controllable face generation method based on facial micro-expressions proposed in the present invention maps facial micro-expressions from discrete vectors to latent codes, and uses the latent codes to fully extract the correlation between facial micro-expressions, so that the model can generate richer facial expression details and improve the quality of the generated results.

[0030] (2) The controllable face generation method based on facial micro-expressions proposed in the present invention, in the above-mentioned convolutional motion generator module, uses adaptive instance normalization (AdaIN) to fuse facial micro-expressions with character images. This allows the model to extract the intrinsic motion details of the character's face using the input facial micro-expressions while keeping the character's appearance features unchanged, so that the model can generate a more realistic character appearance and improve the accuracy of the generated results.

[0031] (3) The controllable face generation method based on facial micro-expressions proposed in the present invention first extracts the face area in the image when constructing the data set, and further screens the image. It is fully guaranteed that the obtained training data set contains face images with different head postures, facial expressions and appearance features, thereby improving the generalization ability of the trained model.

[0032] (4) The controllable face generation method based on facial micro-expressions proposed in the present invention has the ability to artificially control the generation results through parameters. Facial micro-expressions have semantic interpretability, so that our model can intuitively control the generation results by controlling the input facial micro-expression parameters, which has stronger controllability. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of a controllable face generation method based on facial micro-expressions;

[0034] Figure 2 A schematic diagram of the process flow of data set preprocessing of the present invention;

[0035] Figure 3 It is a schematic diagram of the structure of the convolutional coding generator network of the present invention;

[0036] Figure 4 Schematic diagram of the structure of the convolutional motion generator network of the present invention;

[0037] Figure 5 Schematic diagram of the structure of the convolutional image generator network of the present invention; DETAILED DESCRIPTION

[0038] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0039] The controllable face generation method based on facial micro-expressions proposed in the present invention mainly includes two stages in the overall process: training stage and testing stage. The network needs to be trained with real images first. After the training is completed, any face image and specified micro-expression parameters are input into the network, and a face image with a specified expression is output. The generation result can be interpreted and controlled by controlling the input parameters.

[0040] (1) Training phase:

[0041] This stage includes steps 1-4. The convolutional neural network for face generation proposed in the present invention is as follows: Figure 1 The specific contents are as follows:

[0042] Step 1: Input a single training image of size W×H pixels, input the training image into OpenFace for preprocessing, and extract the facial expression coding system vector value in the image Where W and H are both positive integers, and their value range is between 128*128 and 256*256. The facial expression coding system vector ranges from 0 to 5, and is further normalized to compress its value range to between 0 and 1. The extracted features are used as the input micro-expression features F of the training network.

[0043] Step 2: Take the input micro-expression feature F required for training the network obtained in step 1 as the input of the convolutional feature extraction network, and obtain the output feature vector V∈R through a one-dimensional convolutional layer and a skip-connect. M , where M is the latent dimension of the output vector. The latent vector V is further transformed by affine transformation, and the expression information is injected into the subsequent network module through adaptive instance normalization (AdaIN). AdaIN is defined as: AdaIN(x, y) = σ(y)(x-μ(x)) / σ(x)+μ(y). In the formula, μ(·) represents the mean and σ(·) represents the variance.

[0044] Step 3: Compare the output vector V obtained in step 2 with another training image S∈R W×H As the input of the convolutional motion generation network. The output feature map Q∈R is obtained by using the stacked hourglass network (stacked Hourglass Model) W×H , where Q is the feature map used to perform spatial transformation on the source image, and its size is consistent with the input image, which is W×H.

[0045] Step 4: Use the output vector Q obtained in step 3 and the training image S as the input of the convolutional image generation network. The output image is obtained by using the generative adversarial network (Gan) The generator network consists of a downsampling module, a residual module, and an upsampling module, and the discriminator network consists of a downsampling module that acts as an encoder. The final output image is the same size as the input image.

[0046] (2) Testing phase:

[0047] This stage includes steps 5-6, which are as follows:

[0048] Step 5: Combine a single test image of size W × H pixels and a test image of size R 17 The facial micro-expression encoding is used as the input of the network and the facial image with the specified expression is output.

[0049] Step 6: Control the generated results and adjust the generation details by manually controlling the input parameters, and evaluate the model.

[0050] Since the present invention converts facial micro-expressions from discrete vectors to latent codes, and then constructs a convolutional neural network to combine facial micro-expressions with face images, facial micro-expressions are processed as latent vectors in the network, which can preserve the correlation between facial micro-expressions, thereby achieving better image generation effects. At the same time, the present invention retains the ability to artificially control the generation results through parameters, and has stronger controllability.

[0051] Embodiment: The steps of the present invention are described in detail below using the FaceForensics++ dataset as an example.

[0052] Experimental conditions: A computer is selected for network training. The computer is equipped with an Intel(R) processor (3.2GHz) and 32GB random access memory (RAM), Ubuntu 64-bit operating system, and NVIDIA GTX 2080ti (12GB) graphics card; the software environment is the deep learning framework Pytorch.

[0053] Experimental object: FaceForensics++ is a talking video dataset consisting of 1,000 video sequences of news scenes. The data comes from 977 YouTube videos, all of which contain a trackable and unobstructed frontal face. This paper preprocesses the raw data of the dataset, uses face_alignment to crop images of the face area, removes data that does not contain a complete portrait face and has blurred facial expressions, and uniformly scales them to 256*256 pixels.

[0054] Experimental steps:

[0055] (1) Training phase:

[0056] Step 1: Construct the experimental data set. Figure 2 As shown in the figure, each 256*256 face image in the preprocessed image dataset is recognized frame by frame using OpenFace and the facial expression coding system vector value is extracted. Then the training set and the test set are randomly divided into two parts according to a certain ratio, that is, 900*250 images (900 videos, 250 images per video) are randomly selected from the dataset as training data, and the remaining 100*50 images (100 videos, 50 images per video) are used as test data. Finally, a training data generator is created to train the model.

[0057] Step 2: Construct the network. Figure 1 Convolutional neural network is constructed as shown.

[0058] First, the facial expression code is input into the convolutional feature encoder to obtain a size of R 128 The network module consists of 5 layers of one-dimensional convolutional layers (Conv1d) and uses residual links (skip-connect) to retain input information. The specific structure is shown in the attached figure. Figure 3 shown.

[0059] Then the obtained latent code and the face image are input into the convolution motion generator to obtain a size of R 256×256The network module consists of a stacked hourglass network, and uses the adaptive instance normalization (AdaIN) operation after each convolutional layer to inject latent coding information. The encoder network of the stacked hourglass network consists of 5 downsampling modules, and the decoder consists of 5 upsampling modules. The specific structure is shown in the attached figure. Figure 4 shown.

[0060] Finally, the obtained motion feature map and the face image are input into the convolution image generator to obtain the generated image. The size of the generated image is consistent with the input image, which is 256*256 pixels. The network module consists of a generative adversarial network (Gan), in which the generator network (Generator) consists of 2 downsampling modules, 6 residual modules and 2 upsampling modules, and the discriminator network (Discriminator) consists of 4 downsampling modules that act as encoders. The specific structure is shown in the attached figure. Figure 5 shown.

[0061] The network uses the sigmoid activation function in the last layer to obtain the generated result, and the remaining activation functions all use ReLU. The discriminator part of the network uses the instance normalization layer (InstanceNorm), and the remaining normalization layers all use batch normalization (BatchNorm).

[0062] Step 3: Train the network. The Adam (Adaptive Moment Estimation) optimizer was used during training, with an initial learning rate of 2e-4. After 60 iterations, the learning rate was reduced to 2e-5, and a total of 100 iterations were performed. All network parameters were initialized randomly. The batch size for all experiments was set to 20.

[0063] (2) Testing phase:

[0064] The specific test contents are as follows:

[0065] Step 1: Reproduce facial expressions based on real images. Randomly select other face images in the test set and try to reproduce the expressions of the selected images on the test images. Input the test image and the facial expression encoding based on the real image into the trained network to obtain the generated image.

[0066] Step 2: Edit facial expressions intuitively based on parameters. By manually controlling the parameters, the test image can be made to show a specified expression, such as surprise, fear, etc. The test image and the facial expression encoding based on manual control are input into the trained network to obtain the generated image.

[0067] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A controllable face generation method based on facial micro-expressions, characterized in that: Let facial micro-expression F∈R 17 Transformed into latent code V∈R M , where M is the potential dimension obtained by inputting the convolutional feature encoder, and then the face image S∈R W×H The convolution image generator with the potential code V as input generates a specified image, which specifically includes the following steps: (1) Preprocessing of face images; (2) Construct a convolutional neural network driven by facial micro-expressions; (3) Controllable face generation based on facial micro-expression parameters; The size of each face image is W×H, where W is the actual image width and H is the actual image height. Wherein, step (2) of constructing a convolutional neural network driven by facial micro-expressions specifically comprises the following steps: (2-1) Let facial micro-expression F∈R 17 Input to the convolutional feature encoder to obtain the potential vector V∈R M , where M is the potential dimension size obtained by inputting the convolutional neural network. The convolutional feature encoder here contains one-dimensional convolution and residual connection; (2-2) Substitute the latent code V obtained in step (2-1) and the face image S∈R W×H Input to the convolutional motion generator, output feature matrix Q∈R W×H , where W×H is the size of the face image and feature matrix. The convolutional motion generator here contains a stacked hourglass network and adaptive instance normalization; (2-3) The feature matrix Q∈R obtained in step (2-2) W×H and face image S∈R W×H Input to the convolution image generator to get the output image The convolutional image generator here contains a generative adversarial network.

2. The method for generating a controllable human face based on facial micro-expressions according to claim 1, characterized in that: The preprocessing of the face image described in step (1) specifically includes the following steps: (1-1) Use face-alignment to frame and crop the face area in the dataset, and remove face images that do not contain complete portrait faces and have blurred facial expressions. (1-2) Use OpenFace to extract facial micro-expression features F∈R from the face region cropped in step (1-1) 17 , and establish an index according to the data set storage format. F is a vector value based on the facial expression coding system, ranging from 0 to 1; (1-3) The images in the dataset are divided into two parts: training data and test data according to the actual needs in the ratio of 8:

2. D1 face images are randomly selected from the dataset as training data, and the remaining (D-D1) face images are used as test data, where D represents the total number of face images in the dataset, and 0 <D1<D。 3. The method for generating a controllable human face based on facial micro-expressions according to claim 1, characterized in that: The method for controllable face generation based on facial micro-expression parameters described in step (3) is: inputting the face image and the specified facial micro-expression into the convolutional neural network constructed in step (2) for generation, and obtaining the final generation result.

Images