A facial gesture enhanced interactive method based on facial feature transfer
Through the GAN-based Pix2Pix method, combined with U-Net and PatchGAN, the problems of image feature migration accuracy and device requirements in face capture and tracking in the prior art are solved, and high-precision and realistic image conversion are achieved, and the face is improved Capture the system's functional completion degree.
Patent Information
- Application Number
- CN202111042103.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-07
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-09-07
AI Technical Summary
In the field of face capture and tracking, it is difficult to achieve high-precision and realistic image conversion during image feature migration, and the equipment and professional knowledge requirements are high, limiting the application of automatic synthesis and manipulation of facial videos.
Using the Pix2Pix method based on generative adversarial network (GAN), the conditional generative adversarial network modeling is used, and the U-Net network structure and PatchGAN discriminator can be used to realize high-precision image-to-image translation, combining the Dlib library and the OpenCV library for facials. Feature detection and image processing, train generators and discriminators to learn mapping functions to realize the transfer of face features.
The functional completion degree and image conversion accuracy of the face capture system are improved, and high-quality image conversion is achieved through the loss function optimization goal, so that realistic output images can be generated without significantly reducing the authenticity of the image.
Smart Images

Figure CN113610058B_ABST
Abstract
Description
Technical Field
[0001] The invention designs a facial gesture enhanced interaction method for facial feature migration, and belongs to the technical field of virtual reality and augmented reality. Background Art
[0002] Currently, virtual computer animation (CG) in video games and movies is increasingly using face capture and tracking algorithms, which are the basis for the application of augmented reality (AR) and virtual reality (VR) technology. In the field of image processing, facial feature migration is actually the process of converting an input face image into a corresponding output face image. Unlike in the past, the restrictions on professional knowledge, equipment conditions, etc. required in the field of image processing are becoming less and less, and it is becoming easier to automatically synthesize images, facial videos, or manipulate facial images of people. Taking automatic language translation as an example, given enough training data to complete the training of the defined automatic images, in this case, the image conversion method is used to convert one possible representation of the scene into another. In this direction, convolutional neural networks are the best way in the field of image prediction and image feature extraction, and can achieve high-precision data classification functions.
[0003] Convolutional neural networks can actively learn on their own, and in addition they can perform many manual operations, such as adjusting parameters such as loss functions, to make the output images clear and realistic. If you need to set a high target level, requiring the output image to be as close to reality as possible so that it is difficult to distinguish, it is very meaningful to automatically learn the loss function that is suitable for this requirement, and then a generative adversarial network (GAN) was born. If the output image is judged to be true or false, GAN will learn a loss function to classify it while minimizing the loss. Blurred images that are easily seen as fake are not tolerated. GAN can handle data loss well, so it is very suitable for completing tasks that require multiple different types of loss functions as parameters. Summary of the invention
[0004] Purpose of the invention: In order to overcome the deficiencies in the prior art, the present invention provides a facial gesture enhanced interactive method for facial feature transfer, which adopts an image-to-image translation algorithm based on a generative adversarial network to realize a face capture system based on feature transfer. Pix2Pix applies a mapping function of a conditional generative adversarial network modeling conversion to convert an image from one domain to another domain to realize feature transfer. More precisely, Pix2Pix learns from facial markers and converts them into facial features, and places the face detected by a webcam or video on a trained face model in real time.
[0005] Technical solution: To achieve the above purpose, the technical solution adopted by the present invention is:
[0006] A facial gesture enhanced interactive method for facial feature transfer comprises the following steps:
[0007] Step 1, create a data set, use the facial landmark detector in the open source tool Dlib library and the 68 facial landmarks on the faces in the data set as training data using the BSD machine learning software library. Extract each frame of the loaded training video and resize it to convert it into a grayscale image. Create a face detector, landmark predictor, and shape predictor to identify key points of the detected faces and return a sequence of detection elements. Use the Polylines function in the OpenCV library to draw an image that returns the position information and save it as facial features frame by frame to implement the feature extraction function of the system, and finally obtain a data set consisting of landmark images and original images.
[0008] Step 2, train the enhanced model. During the training process, the resized source image and the identification image are merged into a new data set, which is divided into a training set and a validation set. According to the obtained training set, the enhanced model is trained. First, the generator graph and the discriminator graph are obtained, and then the training system starts to work. According to the Epoch and Step, the prediction-true, prediction-false, loss, gradient, variable, discriminator loss, discriminator level and variable, generator GAN loss, generator L1 distance loss, generator level parameters are calculated. The source image and the identification image are merged into an image pair as input, and the mapping function between the input image and the output image is learned, and the trained enhanced model is returned.
[0009] Step 3: Simplify the trained enhanced model to obtain a simplified model whose return size is smaller than the original size, and freeze the simplified model into a single file.
[0010] Step 4, import the simplified model and load it into the memory, take the face captured by the camera as the input image, perform preliminary processing on the input image in the same way as the training model, extract features from the captured face image, obtain the feature identification image, and output the corresponding image in the model, thus realizing the function of migrating the input face features to the face model.
[0011] Preferred: The generator structure in the enhanced model adopts a U-Net network structure. The U-Net network structure includes the left part of the U-Net network, the right part of the U-Net network, and the middle part of the U-Net network. The left part of the U-Net network performs compression operations, reduces the image size through convolution and upsampling, and adds channels to extract shallow features. The right part of the U-Net network performs decoding operations, extracts deep features through convolution and upsampling. The middle part of the U-Net network merges the feature maps obtained from the left and right parts, and refines and predicts the segmentation of the image by combining deep features and shallow features. The U-Net network structure is symmetrical, and skip connections can be added to connect the layers of pixels of the same size with channels.
[0012] Preferably, the discriminator structure in the enhanced model adopts a Markov discriminator, the input of the discriminator is 8N*8N, 8 patches are set, and the output size of the discriminator is N*N, where each value represents the probability that the corresponding 8*8 area in the input image is true.
[0013] Preferably: the Pix2Pix method is used to train the enhancement model. The Pix2Pix method is as follows:
[0014] i) Let x be the edge image of the input image, and y be the source image of the input image. The Pix2Pix method is supervised during training, so paired images (x, y) are required for training. The generator G converts the input x into the generated image G(x), merges x and G(x) together through the channel dimension as the input of the discriminator D, calculates the predicted probability value, and also merges x with the real image y through the channel dimension and inputs it to the discriminator D to obtain the probability prediction value. The predicted probability value indicates the authenticity of the input paired image. The closer the probability value is to 1, the more certain the discriminator D is that the input image pair is real.
[0015] ii) Let z be a noise vector in a random uniform distribution, x be an edge image, and y be the real output image. The generative adversarial network learns the mapping from z to y, G:z→y. The conditional generative adversarial network learns the mapping from x and z to y, G:{x,z}→y. The trained discriminator D is used for identification. The generator G is trained to produce outputs that are as similar as possible to the real image, and ultimately cannot be distinguished by the trained discriminator D. The discriminator D is trained to detect the forgeries produced by the generator G. Therefore, the training goal of the discriminator D is to output a small probability value when the input is not a pair of real images (x, G(x)), and to output a large probability value when the input is a pair of real images (x, y). The training goal of the generator G is to make the probability value output by the discriminator D as large as possible when the generated G(x) and x are used as the input of the discriminator D, which is equivalent to successfully deceiving the discriminator D.
[0016] L c (G,D)=E x,y [logD(x,y)]+E x,z [log(1-D(x,G(x,z))] (1)
[0017] Where, L c (G, D) is the generator G trying to obtain the minimum value of this loss function, E x,y [logD(x,y)] represents the expectation of the image, D(x,y) represents the discriminator feature, E x,z [log(1-D(x,G(x,z))] represents the expectation corresponding to the discriminator feature learning, D(x,G(x,z)) represents the integration of the generator result into the discriminator process, and the discriminator D tries to maximize the loss function value G * ,Right now G represents the discrimination result of the image by the generator, and D represents the discrimination result of the image by the discriminator.
[0018] Preferably, the detection element sequence in step 1 includes mouth features, left eyebrow features, right eyebrow features, nose bridge features, lower nose features, left eye features and right eye features in the facial features and corresponding subscripts.
[0019] Compared with the prior art, the present invention has the following beneficial effects:
[0020] 1. The Pix2Pix method is used to complete the design and implementation of the face capture system based on feature migration, and the system has a high degree of completion of the required functions. The loss function is used to define the effect of the neural network model and the optimization goal, to ensure the accuracy of the training model as much as possible, and "U-Net" is used instead of the encoder-decoder skip connection to achieve the purpose of improving the image-to-image conversion performance.
[0021] 2. Pix2Pix cleverly uses the GAN framework to provide a general framework for problems such as "image to image translation", using U-Net to enhance details and PatchGAN to process the high-frequency parts of the image. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 It is a system framework diagram of the present invention. DETAILED DESCRIPTION
[0023] The present invention is further explained below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.
[0024] A facial gesture enhanced interactive method for facial feature transfer, firstly processes the original data to obtain frames and facial landmark sketches, and then merges the two types of images after resizing to obtain preprocessed data. The processed data is submitted to the conditional generative adversarial network model for image prediction, and finally the migrated result image is generated. This feature transfer process is actually the conversion of the face image of the original data into a face image in the model according to the features. A generative adversarial network-based image-to-image translation algorithm is used to implement a face capture system based on feature transfer. Pix2Pix applies the mapping function of the conditional generative adversarial network modeling conversion to convert images from one domain to another domain to realize feature transfer. More precisely, Pix2Pix learns from facial markers and converts them into facial features, and places the face detected by the webcam or video on the trained face model in real time. Figure 1 As shown, the following steps are included:
[0025] Step 1, create a data set, use the facial landmark detector in the open source tool Dlib library and the BSD machine learning software library to detect 68 facial landmarks on the face in the data set, such as mouth, eyebrows, eyes, etc. as training data. Extract each frame of the loaded training video and resize it to convert it into a grayscale image, create a face detector, landmark predictor and shape predictor to identify the key points of the detected face, and return a sequence of detection elements. The detection element sequence includes the mouth features, left eyebrow features, right eyebrow features, nose bridge features, lower nose features, left eye features and right eye features in the facial features and the corresponding subscripts. Use the Polylines function in the OpenCV library to draw an image with returned position information, save it as frames of facial features, realize the feature extraction function of the system, and finally obtain a data set consisting of a landmark image and an original image.
[0026] Step 2, train the enhanced model. During the training process, the resized source image and the identification image are merged into a new data set, which is divided into a training set and a validation set. According to the obtained training set, the enhanced model is trained. First, the generator graph and the discriminator graph are obtained, and then the training system starts to work. According to the Epoch and Step, the prediction-true, prediction-false, loss, gradient, variable, discriminator loss, discriminator level and variable, generator GAN loss, generator L1 distance loss, generator level parameters are calculated. The source image and the identification image are merged into an image pair as input, and the mapping function between the input image and the output image is learned, and the trained enhanced model is returned.
[0027] The generator structure in the enhanced model uses a U-Net network structure. The U-Net network structure includes the left part of the U-Net network, the right part of the U-Net network, and the middle part of the U-Net network. The left part of the U-Net network performs compression operations (Encoder), reduces the image size through convolution and upsampling, and adds channels to extract shallow features. The right part of the U-Net network performs decoding operations (Decoder), extracts deep features through convolution and upsampling. The middle part of the U-Net network merges the feature maps obtained from the left and right parts, and refines and predicts the segmentation of the image by combining deep features and shallow features. The U-Net network structure is symmetrical, and skip connections can be added to connect the layers of pixels of the same size with channels.
[0028] The discriminator structure in the enhanced model uses a Markov discriminator. The input of the discriminator is 8N*8N. If 8 patches are set, the output size of the discriminator is N*N, and each value represents the probability that the corresponding 8*8 area in the input image is true. This can reduce the input dimension, reduce the number of parameters, reduce the amount of calculation, speed up the training, and remove the restriction on the image size. In addition, because G itself has no restrictions on the image size, the entire Pix2Pix framework has strong scalability.
[0029] The Pix2Pix method is used to train the enhancement model. The Pix2Pix method is as follows:
[0030] i) Let x be the edge image of the input image, and y be the source image of the input image. The Pix2Pix method is supervised during training, so paired images (x, y) are required for training. The generator G converts the input x into the generated image G(x), merges x and G(x) together through the channel dimension as the input of the discriminator D, calculates the predicted probability value, and also merges x with the real image y through the channel dimension and inputs it to the discriminator D to obtain the probability prediction value. The predicted probability value indicates the authenticity of the input paired image. The closer the probability value is to 1, the more certain the discriminator D is that the input image pair is real.
[0031] ii) Let z be a noise vector in a random uniform distribution, x be an edge image, and y be the real output image. The generative adversarial network learns the mapping from z to y, G:z→y. The conditional generative adversarial network learns the mapping from x and z to y, G:{x,z}→y. The trained discriminator D is used for identification. The generator G is trained to produce outputs that are as similar as possible to the real image, and ultimately cannot be distinguished by the trained discriminator D. The discriminator D is trained to detect the forgeries produced by the generator G as much as possible. Therefore, the training goal of the discriminator D is to output a small probability value (for example, the minimum is 0) when the input is not a pair of real images (x, G(x)), and to output a large probability value (for example, the maximum is 1) when the input is a pair of real images (x, y). The training goal of the generator G is to make the probability value output by the discriminator D as large as possible when the generated G(x) and x are used as the input of the discriminator D, which is equivalent to successfully deceiving the discriminator D.
[0032] L c (G,D)=E x,y [logD(x,y)]+E x,z [log(1-D(x,G(x,z))] (1)
[0033] Where, L c (G, D) is the generator G trying to obtain the minimum value of this loss function, E x,y [logD(x,y)] represents the expectation of the image, D(x,y) represents the discriminator feature, E x,z [log(1-D(x,G(x,z))] represents the expectation corresponding to the discriminator feature learning, D(x,G(x,z)) represents the integration of the generator result into the discriminator process, and the discriminator D tries to maximize the loss function value G * ,Right now G represents the discrimination result of the image by the generator, and D represents the discrimination result of the image by the discriminator.
[0034] Step 3, simplify the trained enhanced model to obtain a simplified model with a return size smaller than the original size, and freeze the simplified model into a single file. Simplifying the enhanced model can reduce the amount of model data and optimize the real-time running speed of the system, but it also changes the size of the image and reduces the quality of the generated image.
[0035] Step 4, import the simplified model and load it into the memory, take the face captured by the camera as the input image, perform preliminary processing on the input image in the same way as the training model, extract features from the captured face image, obtain the feature identification image, and output the corresponding image in the model, thus realizing the function of migrating the input face features to the face model.
[0036] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A facial gesture enhanced interactive method for facial feature migration, characterized in that: The steps include: Step 1, create a data set, use the facial landmark detector in the open source tool Dlib library and the 68 facial landmarks on the faces in the data set as training data based on the BSD machine learning software library; extract each frame of the loaded training video and convert it into a grayscale image after resizing, create a face detector, landmark predictor and shape predictor to identify key points of the detected faces, and return a sequence of detection elements; use the Polylines function in the OpenCV library to draw an image with returned position information, save it as facial features frame by frame, realize the feature extraction function of the system, and finally obtain a data set consisting of landmark images and original images; Step 2, training the enhanced model; During the training process, the resized source image and the identification image are merged into a new data set, and the data set is divided into a training set and a validation set. The enhancement model is trained based on the obtained training set. First, the generator graph and the discriminator graph are trained, and then the training system starts to work. According to the Epoch and Step, the prediction-true, prediction-false, loss, gradient, variable, discriminator loss, discriminator level and variable, generator GAN loss, generator L1 distance loss, generator level parameters are calculated. The source image and the identification image are merged into an image pair as input, and the mapping function between the input image and the output image is learned, and the trained enhancement model is returned. The generator structure in the enhanced model uses a U-Net network structure. The U-Net network structure includes the left part of the U-Net network, the right part of the U-Net network, and the middle part of the U-Net network. The left part of the U-Net network performs compression operations, reduces the image size through convolution and upsampling, and adds channels to extract shallow features; the right part of the U-Net network performs decoding operations, extracts deep features through convolution and upsampling; the middle part of the U-Net network merges the feature maps obtained from the left and right parts, and combines deep features with shallow features to refine and predict segmentation of the image; the U-Net network structure is symmetrical, and skip connections can be added to connect the layers of pixels with the same size with channels; The Pix2Pix method is used to train the enhancement model; the Pix2Pix method is as follows: i) Let x be the edge image of the input image, and y be the source image of the input image. The Pix2Pix method is supervised during training, so paired images (x, y) are required for training; the generator G converts the input x into the generated image G(x), merges x and G(x) together through the channel dimension as the input of the discriminator D, calculates the predicted probability value, and also merges x with the real image y through the channel dimension and inputs it to the discriminator D to obtain the probability prediction value; the predicted probability value represents the authenticity of the input paired image, and the closer the probability value is to 1, the more certain the discriminator D is that the input image pair is real; ii) Let z be a noise vector in a random uniform distribution, x be an edge image, and y be a real output image. The generative adversarial network learns the mapping from z to y, G: z → y; while the conditional generative adversarial network learns the mapping from x and z to y, G: {x,z} → y; the trained discriminator D is used for identification, and the generator G is trained to produce outputs that are as similar as possible to the real image, and ultimately cannot be distinguished by the trained discriminator D, and the discriminator D is trained to detect the forgeries produced by the generator G; therefore, the training goal of the discriminator D is to output a small probability value when the input is not a pair of real images (x, G(x)), and to output a large probability value when the input is a pair of real images (x, y); and the training goal of the generator G is to make the probability value output by the discriminator D as large as possible when the generated G(x) and x are used as the input of the discriminator D, which is equivalent to successfully deceiving the discriminator D; (1) In the formula, The generator G tries to obtain the minimum value of this loss function. represents the expectation of the image, represents the discriminator features, represents the expectation corresponding to the discriminator feature learning, Indicates that the results of the generator are integrated into the discriminator process, and the discriminator D tries to maximize the value of this loss function ,Right now , Represents the generator’s discrimination result on the image, Represents the discriminator's discrimination result on the image; Step 3, simplifying the trained enhanced model to obtain a simplified model whose return size is smaller than the original size, and freezing the simplified model into a single file; Step 4, import the simplified model and load it into the memory, take the face captured by the camera as the input image, perform preliminary processing on the input image in the same way as the training model, extract features from the captured face image, obtain the feature identification image, and output the corresponding image in the model, thus realizing the function of migrating the input face features to the face model.
2. The facial gesture enhancement interaction method for facial feature migration according to claim 1, characterized in that: The discriminator structure in the enhanced model uses a Markov discriminator. The input of the discriminator is 8N*8N. If 8 patches are set, the output size of the discriminator is N*N, and each value represents the probability that the corresponding 8*8 area in the input image is true.
3. The facial gesture enhancement interaction method for facial feature migration according to claim 2, characterized in that: The detection element sequence in step 1 includes mouth features, left eyebrow features, right eyebrow features, nose bridge features, lower nose features, left eye features, right eye features and corresponding subscripts in facial features.
