Facial expression transfer method based on adversarial generation and neural radiance fields

Abstract

A face mask expression transfer method based on adversarial generation and neural radiation field, which is composed of the steps of establishing a folk culture face mask data set, constructing a folk culture transfer model, training a stacked hourglass network, training an adversarial generation network, training a neural radiation field network, and testing a folk culture transfer network. By identifying the face mask key points, the face key point recognition range is expanded to the face mask, the face mask key point recognition ability is improved, and the three-dimensional image of the face mask image with the driving video expression characteristics is generated by training the neural radiation field network, which solves the technical problem that the existing expression transfer can only be two-dimensional image, and improves the effect of video face mask expression transfer.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to facial expression transfer in human images. Background Technology

[0002] Since 2017, facial expression transfer technology has gradually entered the public eye. Deepfakes, which swap faces between individuals, have rapidly gained popularity worldwide. Facial expression transfer refers to using deep learning technology to transfer the facial expressions and movements of a person in an original video onto the face of a target person, allowing the target person to make the corresponding expressions. This technology, which applies computer image processing and deep learning, can create many vivid and interesting works and has broad development prospects, attracting many researchers to study facial expression transfer technology. In recent years, facial expression transfer technology has developed rapidly, but its application is often limited to two-dimensional images. The training datasets used for facial expression transfer are mainly composed of images of European and American faces, making it difficult to recognize abstract faces like those depicted on facial masks, resulting in poor transfer performance. Therefore, it is meaningful to develop a model that can effectively transfer facial expressions onto folk cultural elements such as facial masks and allows for multi-angle observation. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a facial expression transfer method based on adversarial generation and neural radiation field that can generate three-dimensional stereoscopic images.

[0004] The technical solution adopted to solve the above technical problems consists of the following steps:

[0005] (1) Establish a dataset of facial features of folk culture

[0006] 1) Obtain publicly available facial images from the internet, save the images as web graphics, crop the facial images into squares, adjust the image size to 256×256 pixels, and perform grayscale processing.

[0007] 2) Manually annotate key points on the facial makeup images to create a folk culture facial makeup image dataset Y, where Y∈{a n ,b n ,c i ,d n ,e n}, where a n Indicates the number of the key storage point, a n ∈{1,2,...,27200},b n This indicates the image number to which the key point belongs, b n ∈{1,2,...,400},c i Indicates the name of the key point, c i∈{1,2,...,64},d n d represents the x-coordinate of the key point. n ∈{1,2,...,256}, e n e represents the ordinate of the key point n ∈{1,2,...,256}; The facial landmark annotation method is used to annotate the facial images, and the landmark data is converted into a character-separated value format for storage.

[0008] 3) Divide the dataset into training, validation, and test sets.

[0009] The facial images were divided into three sets: 80% for training, 15% for validation, and 5% for testing.

[0010] (2) Constructing a folk culture migration network

[0011] The folk culture migration network is composed of a stacked hourglass network, an adversarial generative network, and a neural radiation field network connected in sequence.

[0012] The stacked hourglass network consists of 68 hourglass sub-modules connected in series.

[0013] The adversarial generative network consists of a mapping network and a generative network connected in series. The mapping network consists of eight fully connected layers connected in series. The generative network consists of convolutional modules with a kernel size of 3×3, a padding size of 1, and a stride size of 1.

[0014] The neural radiation field network consists of eight fully connected layers connected in series.

[0015] (3) Training the stacked hourglass network

[0016] 1) Determine the objective function of the stacked hourglass network.

[0017] The loss function MSELoss of the stacked hourglass network is determined by the following formula:

[0018]

[0019] Where i is the x-coordinate of a pixel in the heatmap, j is the y-coordinate of a pixel in the heatmap, and a ij To predict the Gaussian distribution values ​​of key points, b ij The values ​​represent the actual Gaussian distribution values ​​at the key points.

[0020] The activation function relu(x) of the stacked hourglass network is determined by the following formula:

[0021]

[0022] Where x represents the output of the previous layer.

[0023] 2) Input the training set into the stacked hourglass network for training.

[0024] 50% of the training set was input into the stacked hourglass network for training. The Adam optimizer was used, and the learning rate was set to 0.001. Training continued until the loss function MSELoss of the stacked hourglass network converged.

[0025] (4) Training the adversarial generative network

[0026] 1) Determine the objective function of the adversarial generative network.

[0027] The activation function LKrelu(σ) of the adversarial generative network is determined by the following formula:

[0028]

[0029] Where σ is the input to the adversarial generative network, and ε takes the value [0.1, 0.2].

[0030] The cross-entropy loss function f of the adversarial generative network is determined by the following formula:

[0031]

[0032] Among them, gr i Is the i-th image a generated image? gr i The value is 1 when the i-th image is a generated image and 0 when it is not a generated image; gf is the score for judging that the image is not a generated image, and the value of gf is (0,1]; n represents the total number of images, and n is a finite positive integer.

[0033] 2) Input the training set into the adversarial generative network for training, generate the corresponding binary format file of the training set, input the data into the adversarial generative network for training, use the Adam optimizer, set the learning rate to 0.01, and train until the adversarial generative network converges.

[0034] (5) Training neural radiation field network

[0035] 1) Determine the objective function of the neural radiation field network.

[0036] The loss function L1 of the neural radiation field network is determined by the following formula:

[0037]

[0038] Among them, pl k It is the network predicting the radiometric value of the Kth pixel, rl k is the actual radiation value of the Kth pixel, and m represents the total number of pixels in the image, where m is a finite positive integer.

[0039] The activation function relu(α) of the neural radiation field network is determined by the following formula:

[0040]

[0041] Where α represents the output of the previous fully connected layer;

[0042] 2) Creating a training set for neural radiation field networks

[0043] The images output from the adversarial generative network were used to create a dataset for the neural radiation field network. Different facial expressions for each face type were then used as the original training data for the neural radiation field network.

[0044] 3) Input the dataset generated after training the adversarial generative network into the neural radiation field network for training. Input the training set data into the neural radiation field network for training. Train the corresponding 3D image of each face to generate the corresponding 3D image of face expression transfer. Input the validation set into the folk culture transfer network for validation and adjust the model parameters.

[0045] (6) Testing the Folk Culture Migration Network

[0046] The test set was input into the folk culture migration network for testing.

[0047] In the step of constructing the folk culture migration network in (2) of the present invention, the neural radiation field network is composed of 8 fully connected layers connected in series, and each fully connected layer has 256 channels.

[0048] In step 1) of training the adversarial generative network in (4) of the present invention, the optimal value of ε is 0.15. The value of gf is (0,1).

[0049] Compared with the prior art, the present invention has the following advantages:

[0050] This invention employs a combination of stacked hourglass networks, adversarial generative networks, and neural radiation field networks to perform expression transfer on facial representations. The stacked hourglass keypoint detection network is trained to identify facial keypoints, extending the scope of keypoint recognition to the abstract face itself. The neural radiation field network is trained to generate facial images with driving video expression features, enabling better expression transfer onto the face. This overcomes the limitation of existing expression transfer methods, which only support two-dimensional images, by achieving three-dimensional facial images with video expression features. Attached Figure Description

[0051] Figure 1 This is a flowchart of Embodiment 1 of the present invention. Detailed Implementation

[0052] The present invention will be further described below with reference to the accompanying drawings and embodiments, but the scope of protection of the present invention is not limited to the following embodiments.

[0053] Example 1

[0054] like Figure 1 As shown, the facial expression transfer method based on adversarial generation and neural radiation fields in this embodiment consists of the following steps:

[0055] (1) Establish a dataset of facial features of folk culture

[0056] 1) Obtain publicly available facial images from the internet, save the images as web graphics, crop the facial images into squares, adjust the image size to 256×256 pixels, and perform grayscale processing.

[0057] 2) Manually annotate key points on the facial makeup images to create a folk culture facial makeup image dataset Y, where Y∈{a n ,b n ,c i ,d n ,e n}, where a n Indicates the number of the key storage point, a n ∈{1,2,...,27200},b n This indicates the image number to which the key point belongs, b n ∈{1,2,...,400},c i Indicates the name of the key point, c i ∈{1,2,...,64},d n d represents the x-coordinate of the key point. n ∈{1,2,...,256}, e n e represents the ordinate of the key point n ∈{1,2,...,256}; The facial landmark annotation method is used to annotate the facial images, and the landmark data is converted into a character-separated value format for storage.

[0058] 3) Divide the dataset into training, validation, and test sets.

[0059] The facial images were divided into three sets: 80% for training, 15% for validation, and 5% for testing.

[0060] (2) Constructing a folk culture migration network

[0061] The folk culture migration network is composed of a stacked hourglass network, an adversarial generative network, and a neural radiation field network connected in series.

[0062] The stacked hourglass network in this embodiment consists of 68 hourglass sub-modules connected in series.

[0063] The adversarial generative network in this embodiment consists of a mapping network and a generative network connected in series. The mapping network consists of eight fully connected layers connected in series. The generative network consists of convolutional modules with a kernel size of 3×3, a padding size of 1, and a stride size of 1.

[0064] The neural radiation field network in this embodiment consists of eight fully connected layers connected in series, with each fully connected layer having 256 channels.

[0065] (3) Training the stacked hourglass network

[0066] 1) Determine the objective function of the stacked hourglass network.

[0067] The loss function MSELoss of the stacked hourglass network is determined by the following formula:

[0068]

[0069] Where i is the x-coordinate of a pixel in the heatmap, j is the y-coordinate of a pixel in the heatmap, and a ij To predict the Gaussian distribution values ​​of key points, b ij The values ​​represent the actual Gaussian distribution values ​​at the key points.

[0070] The activation function relu(x) of the stacked hourglass network is determined by the following formula:

[0071]

[0072] Where x represents the output of the previous layer.

[0073] 2) Input the training set into the stacked hourglass network for training.

[0074] 50% of the training set was input into the stacked hourglass network for training. The Adam optimizer was used, and the learning rate was set to 0.001. Training continued until the loss function MSELoss of the stacked hourglass network converged.

[0075] (4) Training the adversarial generative network

[0076] 1) Determine the objective function of the adversarial generative network.

[0077] The activation function LKrelu(σ) of the adversarial generative network is determined by the following formula:

[0078]

[0079] Where σ is the input of the adversarial generative network, and ε takes values ​​in the range of [0.1, 0.2]. In this embodiment, ε is 0.15.

[0080] The cross-entropy loss function f of the adversarial generative network is determined by the following formula:

[0081]

[0082] Among them, gr i Is the i-th image a generated image? gr i The value is 1 when the i-th image is a generated image and 0 when it is not a generated image; n represents the total number of images, and n is a finite positive integer; gf is the score for judging that the image is not a generated image, and the value of gf is (0,1]. In this embodiment, gf is 0.5.

[0083] 2) Input the training set into the adversarial generative network for training, generate the corresponding binary format file of the training set, input the data into the adversarial generative network for training, use the Adam optimizer, set the learning rate to 0.01, and train until the adversarial generative network converges.

[0084] (5) Training neural radiation field network

[0085] 1) Determine the objective function of the neural radiation field network.

[0086] The loss function L1 of the neural radiation field network is determined by the following formula:

[0087]

[0088] Among them, pl k It is the network predicting the radiometric value of the Kth pixel, rl k It is the actual radiance value of the Kth pixel, and m represents the total number of pixels in the image.

[0089] The activation function relu(α) of the neural radiation field network is determined by the following formula:

[0090]

[0091] Here, α represents the output of the previous fully connected layer.

[0092] 2) Creating a training set for neural radiation field networks

[0093] The images output from the adversarial generative network were used to create a dataset for the neural radiation field network. Different facial expressions for each face type were then used as the original training data for the neural radiation field network.

[0094] 3) Input the dataset generated after training the adversarial generative network into the neural radiation field network for training.

[0095] The training set data is input into the neural radiation field network for training. For each type of facial mask, a corresponding 3D image of the facial mask is trained, generating a 3D image of the corresponding facial expression transfer. The validation set is input into the folk culture transfer network for validation, and the model parameters are adjusted.

[0096] (6) Testing the Folk Culture Migration Network

[0097] The test set was input into the folk culture migration network for testing.

[0098] A facial expression transfer method based on adversarial generation and neural radiation fields was developed.

[0099] Example 2

[0100] The facial expression transfer method based on adversarial generation and neural radiation fields in this embodiment consists of the following steps:

[0101] (1) Establish a dataset of facial features of folk culture

[0102] The steps are the same as in Example 1.

[0103] (2) Constructing a folk culture migration network

[0104] The steps are the same as in Example 1.

[0105] (3) Training the stacked hourglass network

[0106] The steps are the same as in Example 1.

[0107] (4) Training the adversarial generative network

[0108] 1) Determine the objective function of the stacked hourglass network.

[0109] The loss function MSELoss of the stacked hourglass network is determined by the following formula:

[0110]

[0111] Where i is the x-coordinate of a pixel in the heatmap, j is the y-coordinate of a pixel in the heatmap, and a ij To predict the Gaussian distribution values ​​of key points, b ij The values ​​represent the actual Gaussian distribution values ​​at the key points.

[0112] The activation function relu(x) of the stacked hourglass network is determined by the following formula:

[0113]

[0114] Where x represents the output of the previous layer.

[0115] 2) Input the training set into the stacked hourglass network for training.

[0116] 50% of the training set was input into the stacked hourglass network for training. The Adam optimizer was used, and the learning rate was set to 0.001. Training continued until the loss function MSELoss of the stacked hourglass network converged.

[0117] (4) Training the adversarial generative network

[0118] 1) Determine the objective function of the adversarial generative network.

[0119] The activation function LKrelu(σ) of the adversarial generative network is determined by the following formula:

[0120]

[0121] Where σ is the input of the adversarial generative network, and ε takes values ​​in the range of [0.1, 0.2]. In this embodiment, ε takes the value of 0.1.

[0122] The cross-entropy loss function f of the adversarial generative network is determined by the following formula:

[0123]

[0124] Among them, gr i Is the i-th image a generated image? gr i The value is 1 when the i-th image is a generated image and 0 when it is not a generated image; n represents the total number of images, and n is a finite positive integer; gf is the score for judging that the image is not a generated image, and the value of gf is (0,1]. In this embodiment, gf is 0.1.

[0125] 2) Input the training set into the adversarial generative network for training, generate the corresponding binary format file of the training set, input the data into the adversarial generative network for training, use the Adam optimizer, set the learning rate to 0.01, and train until the adversarial generative network converges.

[0126] The other steps are the same as in Example 1. A facial expression transfer method based on adversarial generation and neural radiation fields is completed.

[0127] Example 3

[0128] The facial expression transfer method based on adversarial generation and neural radiation fields in this embodiment consists of the following steps:

[0129] (1) Establish a dataset of facial features of folk culture

[0130] The steps are the same as in Example 1.

[0131] (2) Constructing a folk culture migration network

[0132] The steps are the same as in Example 1.

[0133] (3) Training the stacked hourglass network

[0134] The steps are the same as in Example 1.

[0135] (4) Training the adversarial generative network

[0136] 1) Determine the objective function of the stacked hourglass network.

[0137] The loss function MSELoss of the stacked hourglass network is determined by the following formula:

[0138]

[0139] Where i is the x-coordinate of a pixel in the heatmap, j is the y-coordinate of a pixel in the heatmap, and a ij To predict the Gaussian distribution values ​​of key points, b ij The values ​​represent the actual Gaussian distribution values ​​at the key points.

[0140] The activation function relu(x) of the stacked hourglass network is determined by the following formula:

[0141]

[0142] Where x represents the output of the previous layer.

[0143] 2) Input the training set into the stacked hourglass network for training.

[0144] 50% of the training set was input into the stacked hourglass network for training. The Adam optimizer was used, and the learning rate was set to 0.001. Training continued until the loss function MSELoss of the stacked hourglass network converged.

[0145] (4) Training the adversarial generative network

[0146] 1) Determine the objective function of the adversarial generative network.

[0147] The activation function LKrelu(σ) of the adversarial generative network is determined by the following formula:

[0148]

[0149] Where σ is the input of the adversarial generative network, and ε takes values ​​in the range of [0.1, 0.2]. In this embodiment, ε takes the value of 0.2.

[0150] The cross-entropy loss function f of the adversarial generative network is determined by the following formula:

[0151]

[0152] Among them, gr i Is the i-th image a generated image? gr iThe value is 1 when the i-th image is a generated image and 0 when it is not; n represents the total number of images, and n is a finite positive integer; gf is the score for judging that the image is not a generated image, and the value of gf is (0,1]. In this embodiment, gf is 1.

[0153] 2) Input the training set into the adversarial generative network for training, generate the corresponding binary format file of the training set, input the data into the adversarial generative network for training, use the Adam optimizer, set the learning rate to 0.01, and train until the adversarial generative network converges.

[0154] The other steps are the same as in Example 1. A facial expression transfer method based on adversarial generation and neural radiation fields is completed.

Claims

1. A facial expression transfer method based on adversarial generation and neural radiation fields, characterized in that... It consists of the following steps: (1) Establish a dataset of facial features of folk culture 1) Obtain publicly available facial images from the internet, save the images as web graphics, crop the facial images into squares, adjust the image size to 256×256 pixels, and perform grayscale processing; 2) Manually annotate key points on the facial makeup images to establish a folk culture facial makeup image dataset. , ,in Indicates the number of the key storage point. , This indicates the image number to which the key point belongs. , Indicates the name of the key point. , This represents the x-coordinate of the key point. , The ordinate of the key point. The facial landmark annotation method is used to annotate the facial images, and the landmark data is converted into a character-separated value format for storage. 3) Divide the dataset into training, validation, and test sets. The facial images were divided into three sets: 80% for training, 15% for validation, and 5% for testing. (2) Constructing a folk culture migration network The folk culture migration network is composed of a stacked hourglass network, an adversarial generative network, and a neural radiation field network connected in sequence. The stacked hourglass network consists of 68 hourglass sub-modules connected in series. The adversarial generative network consists of a mapping network and a generative network connected in series. The mapping network consists of eight fully connected layers connected in series. The generative network consists of convolutional modules with a kernel size of 3×3, a padding size of 1, and a stride size of 1. The neural radiation field network consists of eight fully connected layers connected in series, with each fully connected layer having 256 channels. (3) Training the stacked hourglass network 1) Determine the objective function of the stacked hourglass network. The loss function of the stacked hourglass network is determined by the following formula. : in The x-coordinate of the pixel in the heatmap. The vertical coordinate of the pixel in the heatmap. To predict the Gaussian distribution values ​​of key points, The actual Gaussian distribution values ​​at the key points; The activation function of the stacked hourglass network is determined by the following formula. : in The output of the previous layer is represented; 2) Input the training set into the stacked hourglass network for training. 50% of the training set is fed into the stacked hourglass network for training. The Adam optimizer is used, with a learning rate of 0.

001. The loss function of the stacked hourglass network is then used for training. convergence; (4) Training adversarial generative networks 1) Determine the objective function of the adversarial generative network. The activation function of an adversarial generative network is determined by the following formula. : in, It is the input to the adversarial generative network. The value is 0.15; The cross-entropy loss function of the adversarial generative network is determined by the following formula. : in, Is the i-th image a generated image? The value is 1 when the i-th image is a generated image, and 0 when it is not a generated image. This is the score given for determining that the image is not a generated image. The value of is (0,1]. This represents the total number of images, where n is a finite positive integer; 2) Input the training set into the adversarial generative network for training, generate the corresponding binary format file of the training set, input the data into the adversarial generative network for training, use the Adam optimizer, set the learning rate to 0.01, and train until the adversarial generative network converges. (5) Training neural radiation field network 1) Determine the objective function of the neural radiation field network. The loss function of the neural radiation field network is determined by the following formula. : in, It is the network predicting the radiation value of the Kth pixel. It is the actual radiation value of the Kth pixel. This represents the total number of pixels in the image, where m is a finite positive integer. The activation function of the neural radiation field network is determined by the following formula. : in, This represents the output of the previous fully connected layer; 2) Creating a training set for neural radiation field networks The output images of the adversarial generative network were used to create a dataset for the neural radiation field network. Different facial expressions for each face were generated as the original training data for the neural radiation field network. 3) Input the dataset generated after training the adversarial generative network into the neural radiation field network for training, and input the training set data into the neural radiation field network for training. Train the corresponding 3D image of each face to generate the corresponding 3D image of face expression transfer; input the validation set into the folk culture transfer network for validation and adjust the model parameters. (6) Testing the folk culture migration network The test set was input into the folk culture migration network for testing.

Images