A method and device for face posture transfer based on video drive
Through the two-layer pose migration network model and mask generation network, the deformation and neural network complexity problems of the occlusion part processing in the face pose migration in the prior art are solved, and the high-quality and low-overhead pose migration effect is achieved.
Patent Information
- Application Number
- CN202110892304.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-08-04
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-08-04
AI Technical Summary
The face posture migration method based on the optical flow method in the prior art is prone to deformation and distortion when processing the occluded part. The neural network-based method has too much time overhead due to its complexity and large parameters, which makes it difficult to meet the real-time application requirements. .
The double-layer pose migration network model is adopted, and the high-frequency feature map of face texture features and the low-frequency feature map of face motion trends are fused to generate solid-color background video sequences, and the mask generation network and patching network are used for background repair. Finally, the complete background image is fused with the generated video sequence to achieve the same pose migration as the source image background.
Improves the quality of the pose migration video sequence, reduces time overhead, enhances the usability and ease of use of the method, and enables the generation of high-quality videos that match the source image.
Smart Images

Figure CN113808005B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of digital image processing, and in particular relates to a method and device for migrating facial postures based on video driving. Background Art
[0002] Traditional face pose editing and migration methods are based on computer graphics technology, which generates corresponding mesh or point cloud data for facial poses in images, or uses preset 3D face models to operate on meshes or 3D point clouds to achieve facial pose editing and migration. With the development of deep learning technology, many deep learning network structures have been proven to have excellent performance in tasks such as image generation and style migration, resulting in the emergence of a large number of face pose migration methods based on neural networks and deep learning technologies in recent years.
[0003] The prior art records "a method for facial expression transfer based on self-supervised learning and generative adversarial mechanism". This method uses a self-supervised learning method and a generative adversarial mechanism to decouple the facial identity information and posture information in the image, and inputs the global posture deformation information in the driving video into the network model in combination with the identity information of the target face for image reconstruction, and finally synthesizes the generated continuous video frame sequence into a video. The prior art also records "a two-stage expression animation generation method based on a dual generative adversarial network". This method divides the expression animation production process into two stages, which are implemented using generative adversarial networks respectively. In the first stage, the expression transfer network is used to extract expression features and transfer them, and in the second stage, the detail generation network is used to supplement the generation results of the first stage in a fine-grained manner to achieve a high-quality generation effect. In the "a method and system for expression transfer and animation" disclosed in the prior art, the posture transfer is performed according to the mapping relationship between the target face and the facial key points in each frame of the driving video, that is, by tracking the sparse optical flow of the key points.
[0004] In the "face pose transfer method based on video drive", the estimation of face motion and the generation of images are the key factors to determine the quality of the final generated results of the method. One of the main existing methods is to use the optical flow method to extract the motion information of the face of the character in the driving video, and apply the extracted optical flow to the corresponding face key points of the source image to achieve the driving effect. The advantage of this method is that the processing speed is fast, and it can even meet the needs of real-time face pose transfer. However, there are many limitations to using only the optical flow method, such as the need to use algorithms to repair the parts that are blocked in the source image (such as the side face, back of the head, etc.). However, the current repair algorithm often causes deformation and distortion in the final repair result, and it is difficult to make the repair result and the corresponding part of the source image reach the same resolution and authenticity, so as to achieve the effect that human vision is difficult to distinguish between true and false. Another method is to generate an image with the same target pose as the source image based entirely on a neural network. This method uses one or more trained encoders to encode images and driving video frame sequences, extract feature information of different semantics and dimensions, and then process and splice the feature information, decode it through a specific decoder network, and directly obtain images and videos with the corresponding poses of the target object. The problem with this type of method is that good results often require a more complex network structure and a larger number of parameters, which results in greater time overhead, making it difficult to apply in actual application scenarios. Summary of the invention
[0005] In order to solve the above problems existing in the prior art, the present invention provides a method and device for face posture migration based on video drive. The technical problem to be solved by the present invention is achieved through the following technical solutions:
[0006] In a first aspect, the present invention provides a video-driven face posture migration method comprising:
[0007] Get source images and drive video sequences;
[0008] Wherein, the source image and the driving video sequence contain human faces;
[0009] Extracting first facial key point information from the source image and extracting second facial key point information from the driving video sequence;
[0010] Preprocessing the driving video sequence to obtain a facial motion track coverage area formed by facial motion in the driving video sequence;
[0011] Cropping the area covered by the face motion trajectory in the driving video sequence to obtain a face sub-image sequence;
[0012] Input the first facial key point information and the source image into a first input end of a preset double-layer posture transfer network model, and input the second facial key point information and the facial sub-image sequence into a second input end of the double-layer posture transfer network model, so that the double-layer posture transfer network model extracts a high-frequency feature map containing facial texture features in the source image, and extracts transformation parameters representing facial motion trends and low-frequency feature maps containing contour and color features in the facial sub-image sequence;
[0013] The high-frequency feature map is transformed using the transformation parameters, and the transformed high-frequency feature map is fused with the low-frequency feature map to obtain a generated video sequence with a pure color background;
[0014] Inputting the generated video sequence and the source image into a preset mask generation network respectively to obtain a first mask of the generated video sequence and a second mask of the source image;
[0015] Using the second mask to deduct the background in the source image to obtain a missing image containing only the background;
[0016] Inputting the second mask and the missing image into a preset inpainting network, so that the inpainting network inpaints the missing image to obtain a complete background image of the source image;
[0017] The complete background image is fused with the generated video sequence, so that the complete background image replaces the background part in the generated video sequence according to the first mask, and a posture transfer video sequence with the same background as the source image is obtained.
[0018] Optionally, preprocessing the driving video sequence to obtain a facial motion track coverage area formed by facial motion in the driving video sequence includes:
[0019] Detecting the face position of each frame in the driving video sequence, and obtaining the face area determined by the upper left vertex and the lower right vertex in each frame;
[0020] For each current frame in the driving video sequence, the IOU value of the face area of the current frame and the face track area formed by the previous frame is calculated in frame sequence starting from the first frame. When the IOU value is greater than a threshold, it is determined that the face area of the current frame and the face track area formed by the previous frame belong to the same target face, and the face area of the current frame is merged into the face track area formed by the previous frame until the last frame is completed, thereby obtaining the face motion track coverage area formed by the face motion in the driving video sequence.
[0021] Optionally, the step of merging the face area of the current frame into the face track area formed by the previous frame includes:
[0022] Calculate the face area of the current frame and the face track area formed by the previous frame, the minimum value of the upper left vertex and the maximum value of the lower right vertex;
[0023] The face track area formed by the maximum value and the minimum value is determined as the face area of the current frame and is merged into the face track area formed in the previous frame.
[0024] Among them, the preset two-layer posture transfer network model includes: an encoder, a high-frequency feature decoder and a low-frequency feature decoder, the encoder is connected to the high-frequency feature decoder and the low-frequency feature decoder, and the input end of the encoder includes the first input end and the second input end.
[0025] Optionally, the first facial key point information and the source image are input into a first input end of a preset double-layer posture transfer network model, and the second facial key point information and the facial sub-image sequence are input into a second input end of the double-layer posture transfer network model, so that the double-layer posture transfer network model extracts a high-frequency feature map containing facial texture features in the source image, and extracts transformation parameters representing facial motion trends and low-frequency feature maps containing contour and color features in the facial sub-image sequence; the high-frequency feature map is transformed using the transformation parameters, and the transformed high-frequency feature map is merged with the low-frequency feature map to obtain a generated video sequence with a pure color background, including:
[0026] The first facial key point information and the source image are input into the first input end of a preset double-layer posture transfer network model, and the facial sub-image sequence and the second facial key point information are input into the second input end of the double-layer posture transfer network model, so that the encoder in the double-layer posture transfer network model extracts a first feature vector of the source image based on the first facial key point information and extracts a second feature vector of the facial sub-image sequence based on the second facial key point information; the high-frequency feature decoder decodes the first feature vector to obtain a high-frequency feature map containing facial texture features in the source image; and the low-frequency feature decoder decodes the second feature vector to obtain transformation parameters representing facial motion trends in the facial sub-image sequence and a low-frequency feature map containing contour and color features.
[0027] Optionally, the patching network includes: an encoder, a hierarchical fully connected layer and a decoder, wherein the output of the encoder is connected to the hierarchical fully connected layer, and the output of the hierarchical fully connected layer is connected to the decoder.
[0028] Optionally, inputting the second mask and the missing image containing only the background into a preset inpainting network so that the inpainting network inpaints the missing image to obtain a complete background image of the source image includes:
[0029] The second mask and the missing image are input into an encoder in a preset inpainting network, so that the encoder extracts high-dimensional features of the missing image according to the second mask, the hierarchical fully connected layer guides the missing image to perform background repair and generation according to the high-dimensional features, and the decoder upsamples the generated missing image to restore the missing image to its original size, thereby obtaining a complete background image of the source image.
[0030] Optionally, fusing the complete background image with the generated video sequence so that the complete background image replaces the background portion in the generated video sequence according to the first mask, and obtaining a posture transfer video sequence with the same background as the source image includes:
[0031] The complete background image is fused with the generated video sequence using a Poisson fusion algorithm, so that the complete background image replaces the background part in the generated video sequence according to the first mask, and a posture transfer video sequence identical to the source background is obtained.
[0032] In a second aspect, the present invention provides a video-driven face posture migration device comprising:
[0033] An acquisition module, used to acquire source images and drive video sequences;
[0034] Wherein, the source image and the driving video sequence contain human faces;
[0035] An information extraction module, used to extract first facial key point information from the source image and second facial key point information from a driving video sequence;
[0036] A preprocessing module, used for preprocessing the driving video sequence to obtain a facial motion track coverage area formed by facial motion in the driving video sequence;
[0037] A cropping module, used for cropping the area covered by the face motion trajectory in the driving video sequence to obtain a face sub-image sequence;
[0038] A feature extraction module, used for inputting the first facial key point information and the source image into a first input end of a preset double-layer posture transfer network model, and inputting the second facial key point information and the facial sub-image sequence into a second input end of the double-layer posture transfer network model, so that the double-layer posture transfer network model extracts a high-frequency feature map containing facial texture features in the source image, and extracts transformation parameters representing facial motion trends and a low-frequency feature map containing contour and color features in the facial sub-image sequence;
[0039] A transformation module, used to transform the high-frequency feature map using the transformation parameters, and fuse the transformed high-frequency feature map with the low-frequency feature map to obtain a generated video sequence with a pure color background;
[0040] A generating module, used for inputting the generated video sequence and the source image into a preset mask generating network respectively, to obtain a first mask of the generated video sequence and a second mask of the source image;
[0041] A subtraction module, configured to use the second mask to subtract the background in the source image to obtain a missing image containing only the background;
[0042] A patching module, used for inputting the second mask and the missing image into a preset patching network, so that the patching network patches the missing image to obtain a complete background image of the source image;
[0043] A fusion module is used to fuse the complete background image with the generated video sequence so that the complete background image replaces the background part in the generated video sequence according to the first mask to obtain a posture transfer video sequence with the same background as the source image.
[0044] Optionally, the preprocessing module is specifically used to:
[0045] Detecting the face position of each frame in the driving video sequence, and obtaining the face area formed by the upper left vertex and the lower right vertex in each frame;
[0046] For each current frame in the driving video sequence, the IOU value of the face area of the current frame and the face track area formed by the previous frame is calculated starting from the first frame in accordance with the frame sequence. When the IOU value is greater than the threshold, it is determined that the face area of the current frame and the face track area formed by the previous frame of the current frame belong to the same target face, and the face area of the current frame is merged into the face track area formed by the previous frame of the current frame until the last frame is completed, thereby obtaining the face motion track coverage area formed by the face motion in the driving video sequence.
[0047] The present invention provides a method and process for face posture transfer based on video drive, which pre-processes the driving video sequence to obtain the area covered by the face motion trajectory and then crops it. The cropped face sub-image sequence and the source image are input into a double-layer posture transfer network model to obtain high-frequency information, posture transformation parameters and low-frequency information. The posture transformation parameters are used to transform the high-frequency information and fuse the low-frequency information to obtain a generated video sequence with a pure color background. A mask generation network is then used to generate a mask, and the mask is used to deduct the background in the source image to obtain a missing image containing only the background. The missing image and the mask are input into a patching network to patch the missing image to obtain a complete background image of the source image, and the complete background image is fused with the generated video sequence to obtain a posture transfer video sequence with the same background as the source image. Therefore, the present invention can improve the quality of the posture transfer video sequence, and has better usability and ease of use.
[0048] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1 It is a flow chart of a method for face posture transfer based on video driving provided by an embodiment of the present invention;
[0050] Figure 2 Schematic diagram of the overall process of a video-driven face posture migration method provided by an embodiment of the present invention;
[0051] Figure 3 It is a schematic diagram of the process of preprocessing the driving video sequence provided by the present invention;
[0052] Figure 4 It is a structural diagram of the double-layer posture transfer network model provided by the present invention;
[0053] Figure 5 is a structural diagram of a repair network provided by the present invention;
[0054] Figure 6 It is a structural schematic diagram of a video-driven face posture migration device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0055] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.
[0056] like Figure 1 As shown, a video-driven face posture migration method provided by the present invention includes:
[0057] S1, obtain source image and drive video sequence;
[0058] Wherein, the source image and the driving video sequence contain human faces;
[0059] S2, extracting the first facial key point information from the source image and extracting the second facial key point information from the driving video sequence;
[0060] S3, preprocessing the driving video sequence to obtain a facial motion trajectory coverage area formed by facial motion in the driving video sequence;
[0061] For any image, due to differences in the resolution, focal length, object distance, framing and composition of the acquisition equipment, the position of the face area in the image and the proportion of the image area are often different. Directly processing the entire image will introduce a large number of unstructured errors, the model training is not easy to converge, and the processing effect for new samples is not good, and the generalization performance is poor. Therefore, the two-layer posture transfer network first performs face detection on the input image, crops the detected face area, and then sends the cropped square area to the model for processing to solve the above problems.
[0062] S4, cropping the area covered by the face motion trajectory in the driving video sequence to obtain a face sub-image sequence;
[0063] S5, inputting the first facial key point information and the source image into the first input end of a preset double-layer posture transfer network model, and inputting the second facial key point information and the facial sub-image sequence into the second input end of the double-layer posture transfer network model, so that the double-layer posture transfer network model extracts a high-frequency feature map containing facial texture features in the source image, and extracts transformation parameters representing facial motion trends in the facial sub-image sequence and a low-frequency feature map containing contour and color features;
[0064] Among them, the facial key point information is the information describing the local and global features of the face, and the face can be identified through this information.
[0065] S6, transforming the high-frequency feature map using the transformation parameter, fusing the transformed high-frequency feature map with the low-frequency feature map to obtain a generated video sequence with a pure color background;
[0066] S7, inputting the generated video sequence and the source image into a preset mask generation network respectively to obtain a first mask of the generated video sequence and a second mask of the source image;
[0067] Using the mask generation network, the input is the pure color background video of the generated video sequence and the source image, and the first mask and the second mask are generated. The mask is a binary image. The background of the source image is deducted using the binary image. The obtained second mask is used to deduct the background area with clear boundaries in the generated video sequence, and then the background repair algorithm is used to repair the background area to obtain a complete background image.
[0068] S8, using the second mask to deduct the background in the source image to obtain a missing image containing only the background;
[0069] S9, inputting the second mask and the missing image into a preset inpainting network, so that the inpainting network inpaints the missing image to obtain a complete background image of the source image;
[0070] S10, fusing the complete background image with the generated video sequence, so that the complete background image replaces the background part in the generated video sequence according to the first mask, and obtaining a posture transfer video sequence with the same background as the source image.
[0071] like Figure 2 As shown, Figure 2 The overall process diagram of a video-driven face posture transfer method provided by the present invention is shown in FIG. Figure 2 The first layer of the two-layer posture transfer network model, namely the high-frequency decoder network, is used to extract the high-frequency information of the source image, which only needs to be calculated once in the model inference stage; the second layer of the network, namely the low-frequency feature decoder network, is used to extract the low-frequency information of the face sub-image sequence and predict a set of posture transformation parameters. Finally, the posture transformation parameters of the face sub-image sequence are used to deform the high-frequency information, and then the low-frequency information is fused to generate the image of the source image character in the corresponding posture in the current driving video frame. At the same time, the learning algorithm based on meta-learning is used to achieve better reconstruction effect for the situation where the occluded part of the source image appears in the subsequent generated picture, so that the deformation and jitter of the generated result are reduced.
[0072] The double-layer posture transfer network model of the present invention can realize fast, accurate and high-quality posture transfer between images. In order to expand it to the application of face posture transfer based on video drive. The present invention has made certain expansions and improvements. On the basis of the face posture transfer algorithm, the pre-processing and post-processing methods such as face activity area positioning, video stabilization, and background restoration are added to form a complete set of video-driven image animation process. The whole process can generate higher quality video results with better usability and ease of use.
[0073] The present invention provides a method for face posture transfer based on video drive, which pre-processes the driving video sequence to obtain the area covered by the face motion trajectory and then crops it, inputs the cropped face sub-image sequence and the source image into a double-layer posture transfer network model to obtain high-frequency information, posture transformation parameters and low-frequency information, uses the posture transformation parameters to transform the high-frequency information and fuses the low-frequency information to obtain a generated video sequence with a pure color background. Then, a mask generation network is used to generate a mask, and the background in the source image is deducted using the mask to obtain a missing image containing only the background. The missing image and the mask are input into a preset patching network to patch the missing image, obtain a complete background image of the source image, and fuse the complete background image with the generated video sequence to obtain a posture transfer video sequence with the same background as the source image. Therefore, the present invention can improve the quality of the posture transfer video sequence, and has better usability and ease of use.
[0074] As an optional implementation manner of the present invention, preprocessing the driving video sequence to obtain a face motion track coverage area formed by face motion in the driving video sequence includes:
[0075] Step a: Detect the face position of each frame in the driving video sequence, and obtain the face area formed by the upper left vertex and the lower right vertex in each frame;
[0076] Step b: For each current frame in the driving video sequence, start from the first frame and calculate the IOU value of the face area of the current frame and the face track area formed by the previous frame in the frame sequence. When the IOU value is greater than the threshold, it is determined that the face area of the current frame and the face track area formed by the previous frame belong to the same target face. The face area of the current frame is merged into the face track area formed by the previous frame until the last frame is completed, and the face motion track coverage area formed by the face motion in the driving video sequence is obtained.
[0077] For a sequence of continuous video frames, although face detection for each frame can obtain a square area with a relatively fixed proportion and position of the face area, the generated result will have a lot of jitter, which seriously affects the smoothness and authenticity of the generated video. The reason for this phenomenon is that this strategy performs separate face detection on each frame, and the final result is that the lens will follow the movement of the face posture, while the video-driven posture transfer task requires the lens to be fixed to shoot the change of the face posture. Therefore, it is necessary to examine the motion trajectory of the face area in the entire video to determine the smallest square area that covers all face motion trajectories. This step provides a preprocessing process for detecting the area covered by the face motion trajectory in the video to determine the smallest square area that contains all face motion trajectories.
[0078] As an optional implementation of the present invention, merging the face region of the current frame into the face track region formed by the previous frame includes:
[0079] Step a: Calculate the face area of the current frame and the face track area formed by the previous frame, the minimum value of the upper left vertex and the maximum value of the lower right vertex;
[0080] Step b: The face track area formed by the maximum value and the minimum value is determined as the face area of the current frame and is merged into the face track area formed in the previous frame.
[0081] refer to Figure 3 , the preprocessing process uses the existing face detection library to detect the face area in each frame of video frame by frame. The detected face area is represented by a square area determined by the two coordinates of the upper left corner and the lower right corner. For the newly detected face area in the next frame of video, the algorithm first calculates the IOU value of the current face track area and the newly detected face area, that is, the ratio of the intersection area of the two detected face areas to the total area covered by the two face areas. When the ratio is greater than the set threshold, it is considered that the two detected faces belong to the same target face, and the newly detected face area should be merged into the current face track area. Update the current face track area to the minimum rectangular area covering the original face area and the newly detected face area. The updating method is to calculate the minimum value of the upper left corner coordinates and the maximum value of the lower right corner coordinates of the two face areas. Repeat the above process iteratively, process the video frame by frame, and finally obtain the face motion track area, which is the minimum face motion track coverage area in the entire video. By cropping this area in each frame of the video, a face sub-image sequence that meets the requirements can be obtained.
[0082] refer to Figure 4 The preset two-layer posture transfer network model includes: an encoder, a high-frequency feature decoder and a low-frequency feature decoder. The encoder is connected to the high-frequency feature decoder and the low-frequency feature decoder. The input end of the encoder includes a first input end and a second input end.
[0083] The present invention inputs first facial key point information and a source image into a first input end of a preset double-layer posture transfer network model, and inputs a facial sub-image sequence and second facial key point information into a second input end of the double-layer posture transfer network model, so that an encoder in the double-layer posture transfer network model extracts a first feature vector of the source image based on the first facial key point information and extracts a second feature vector of the facial sub-image sequence based on the second facial key point information, a high-frequency feature decoder decodes the first feature vector to obtain a high-frequency feature map containing facial texture features in the source image; and a low-frequency feature decoder decodes the second feature vector to obtain transformation parameters representing facial motion trends in the facial sub-image sequence and a low-frequency feature map containing contour and color features.
[0084] The encoder receives the first facial key point information of the target pose and the source image as the input of the first end, and the facial sub-image sequence and the second facial key point information as the input of the second end. The input image is downsampled to 8×8 resolution through 5 downsampling modules composed of 2D convolution and average pooling layers. The downsampled result is then sent to 6 equal-resolution network modules to reduce the number of channels in the input while keeping the resolution unchanged. Each equal-resolution network module outputs a vector mapping to initialize the network adaptive parameters inside each decoder. The downsampling module and the equal-resolution module of the encoder network both introduce a skip connection mechanism to avoid the problems of gradient explosion and gradient disappearance, and finally obtain the first eigenvector and the second eigenvector.
[0085] The high-frequency feature decoder decodes the first feature vector output by the encoder to obtain the high-frequency components of the restored source image. The high-frequency part contains the texture features of the image. The convolution kernel parameters of the convolution layer in the high-frequency feature decoder are predicted by the corresponding vector mapping output by the encoder. This strategy can save space by eliminating additional networks and intermediate feature maps to increase the batch size, improve the training speed and training effect of the model. The input of the high-frequency feature decoder is processed by 6 upsampling network modules to obtain the high-frequency feature map of the source image. Each upsampling module also introduces a jump connection mechanism.
[0086] The low-frequency feature encoder decodes the second feature vector to obtain a set of transformation parameters and a low-frequency feature map containing facial features, skin color, and lighting information. The transformation parameters establish a mapping from the high-frequency feature space to the target posture. During the training phase, the input target posture key points are fed into the network in the form of a graph, while during the test phase, the low-frequency feature encoder directly receives the target posture key points in the form of a vector as input. The input of the low-frequency feature encoder is processed by 6 upsampling network modules, each of which also uses a skip connection mechanism. The final output of the upsampling module is activated by different activation functions to obtain the low-frequency components and transformation parameters of the output image.
[0087] refer to Figure 5 The patching network of the present invention may include: an encoder, a hierarchical fully connected layer and a decoder, the output of the encoder is connected to the hierarchical fully connected layer, and the output of the hierarchical fully connected layer is connected to the decoder.
[0088] As an optional implementation of the present invention, inputting the second mask and the missing image containing only the background into a preset inpainting network so that the inpainting network inpaints the missing image, and obtaining a complete background image of the source image includes:
[0089] The second mask and the missing image containing only the background are input into the encoder in the preset inpainting network, so that the encoder extracts the high-dimensional features of the missing image according to the second mask, the hierarchical fully connected layer guides the missing image to perform background repair and generation according to the high-dimensional features, and the decoder upsamples the generated missing image to restore the missing image to its original size to obtain the complete background image of the source image.
[0090] It is understandable that the reconstruction of the image background in the pose transfer task is a difficult problem. Since the backgrounds of different images and videos are often different, especially in the training data set, it takes a lot of manpower and material resources to collect data with the same background. Training with the same background data set will also have a certain impact on the generalization performance of the model. However, learning the static background patterns of different videos in the modeling data set requires a lot of capacity of the high-frequency feature decoder and the low-frequency feature decoder. Moreover, the presence of the background will cause deviations in the network's learning of other features (such as identity information). Therefore, the two-layer pose transfer network uses a face and torso mask generation model to obtain the mask of the segmented head, torso and background areas of the input image and video frame. The mask can well remove the interference of the static background on the model during the training process, ensuring that the learned model has a good reconstruction effect on the head and torso of the person. However, with the introduction of masks in the model training stage, the model's capabilities will focus on maintaining the consistency of reconstruction of the head and torso of the character, and the transformation parameters output by the low-frequency feature decoder are also transformed for the head area, resulting in the background often being distorted after the transformation parameters are applied to the entire image, which has a certain impact on the authenticity of the generated results. There have been a lot of research results on the background repair problem in the field of computer vision, but traditional image completion methods are not effective for images with large missing areas and complex missing content, such as those involved in the application scenarios of the present invention. Therefore, the present invention also introduces a background repair algorithm. The present invention uses a convolutional neural network to complete the background image. The network structure diagram is shown in the figure. Figure 4 As shown in the figure, the network consists of three parts: encoder, channel-level fully connected layer and decoder.
[0091] The network uses the powerful extraction capability of convolutional neural networks for high-dimensional information to learn high-dimensional features in the input image, and guides the restoration and generation of missing parts of the image based on the extracted high-dimensional features. The encoder network draws on the convolutional network structure of the first five layers of AlexNet. The channel-level fully connected layer improves the ordinary fully connected layer by removing the information exchange between feature layers of the ordinary fully connected layer, thereby reducing the parameter scale, and instead uses a convolutional layer after the fully connected layer to achieve inter-layer information exchange. Finally, the decoder upsamples the feature map and restores it to its original size to obtain a complete background image.
[0092] As an optional implementation of the present invention, the complete background image is merged with the generated video sequence so that the complete background image replaces the background part in the generated video sequence according to the first mask, and the posture transfer video sequence with the same source background is obtained, which includes:
[0093] The complete background image is fused with the generated video sequence using a Poisson fusion algorithm, so that the complete background image replaces the background part in the generated video sequence according to the first mask, and a posture transfer video sequence with the same background as the source image is obtained.
[0094] Finally, the Poisson fusion algorithm is combined with the existing video frame mask to replace the distorted background output by the posture transfer network with the complete background image to obtain a posture transfer video with the same background as the source image.
[0095] like Figure 6 As shown, a video-driven face posture migration device provided by the present invention includes:
[0096] An acquisition module 61 is used to acquire a source image and drive a video sequence;
[0097] Wherein, the source image and the driving video sequence contain human faces;
[0098] An information extraction module 62, configured to extract first facial key point information from a source image and drive a video sequence to extract second facial key point information;
[0099] A preprocessing module 63, used for preprocessing the driving video sequence to obtain a facial motion track coverage area formed by facial motion in the driving video sequence;
[0100] A cropping module 64, used for cropping the area covered by the face motion trajectory in the driving video sequence to obtain a face sub-image sequence;
[0101] The feature extraction module 65 is used to input the first facial key point information and the source image into the first input end of the preset double-layer posture transfer network model, and input the second facial key point information and the facial sub-image sequence into the second input end of the double-layer posture transfer network model, so that the double-layer posture transfer network model extracts the high-frequency feature map containing facial texture features in the source image, and extracts the transformation parameters representing the facial motion trend and the low-frequency feature map containing contour and color features in the facial sub-image sequence;
[0102] A transformation module 66, configured to transform the high-frequency feature map using the transformation parameters, and fuse the transformed high-frequency feature map with the low-frequency feature map to obtain a generated video sequence with a pure color background;
[0103] A generating module 67, used to input the generated video sequence and the source image into a preset mask generating network respectively, to obtain a first mask of the generated video sequence and a second mask of the source image;
[0104] A subtraction module 68, configured to subtract the background in the source image using the second mask to obtain a missing image containing only the background;
[0105] A patching module 69 is used to input the second mask and the missing image into a preset patching network, so that the patching network patches the missing image to obtain a complete background image of the source image;
[0106] The fusion module 610 is used to fuse the complete background image with the generated video sequence, so that the complete background image replaces the background part in the generated video sequence according to the first mask to obtain the same posture transfer video sequence as the source background.
[0107] As an optional implementation manner of the present invention, the preprocessing module is specifically used for:
[0108] Detect the face area of each frame in the driving video sequence, and obtain the face area formed by the upper left vertex and the lower right vertex in each frame;
[0109] For each current frame in the driving video sequence, the IOU value of the face area of the current frame and the face track area formed by the previous frame is calculated in frame order starting from the first frame. When the IOU value is greater than the threshold, it is determined that the face area of the current frame and the face track area formed by the previous frame belong to the same target face. The face area of the current frame is merged into the face track area formed by the previous frame until the last frame is completed, and the face motion track coverage area formed by the face motion in the driving video sequence is obtained.
[0110] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.
Claims
1. A video-driven face posture transfer method, characterized in that: include: Get source images and drive video sequences; Wherein, the source image and the driving video sequence contain human faces; Extracting first facial key point information from the source image and extracting second facial key point information from the driving video sequence; The driving video sequence is preprocessed to obtain the face motion track coverage area formed by the face motion in the driving video sequence; including: detecting the face position of each frame in the driving video sequence, and obtaining the face area determined by the upper left vertex and the lower right vertex in each frame; for each current frame in the driving video sequence, starting from the first frame, calculating the IOU value of the face area of the current frame and the face track area formed by the previous frame in accordance with the frame sequence, when the IOU value is greater than a threshold, it is determined that the face area of the current frame and the face track area formed by the previous frame belong to the same target face, and the face area of the current frame is merged into the face track area formed by the previous frame until the last frame is completed, and the face motion track coverage area formed by the face motion in the driving video sequence is obtained; Cropping the area covered by the face motion trajectory in the driving video sequence to obtain a face sub-image sequence; A preset double-layer posture transfer network model is obtained, comprising: an encoder, a high-frequency feature decoder and a low-frequency feature decoder, wherein the encoder is connected to the high-frequency feature decoder and the low-frequency feature decoder, and the input end of the encoder comprises a first input end and a second input end; the first facial key point information and the source image are input into the first input end of the preset double-layer posture transfer network model, and the second facial key point information and the facial sub-image sequence are input into the second input end of the double-layer posture transfer network model, so that the encoder in the double-layer posture transfer network model extracts a first feature vector of the source image based on the first facial key point information and extracts a second feature vector of the facial sub-image sequence based on the second facial key point information, the high-frequency feature decoder decodes the first feature vector to obtain a high-frequency feature map containing facial texture features in the source image; the low-frequency feature decoder decodes the second feature vector to obtain a transformation parameter representing a facial motion trend in the facial sub-image sequence and a low-frequency feature map containing contour and color features; The high-frequency feature map is transformed using the transformation parameters, and the transformed high-frequency feature map is fused with the low-frequency feature map to obtain a generated video sequence with a pure color background; Inputting the generated video sequence and the source image into a preset mask generation network respectively to obtain a first mask of the generated video sequence and a second mask of the source image; Using the second mask to deduct the background in the source image to obtain a missing image containing only the background; Inputting the second mask and the missing image into a preset inpainting network, so that the inpainting network inpaints the missing image to obtain a complete background image of the source image; The complete background image is fused with the generated video sequence, so that the complete background image replaces the background part in the generated video sequence according to the first mask, and a posture transfer video sequence with the same background as the source image is obtained.
2. The method for facial posture transfer according to claim 1, characterized in that: The step of merging the face area of the current frame into the face track area formed by the previous frame comprises: Calculate the face area of the current frame and the face track area formed by the previous frame, the minimum value of the upper left vertex and the maximum value of the lower right vertex; The face track area formed by the maximum value and the minimum value is determined as the face area of the current frame and is merged into the face track area formed in the previous frame.
3. The face posture transfer method according to claim 1, characterized in that: The patching network includes: an encoder, a hierarchical fully connected layer and a decoder, wherein the output of the encoder is connected to the hierarchical fully connected layer, and the output of the hierarchical fully connected layer is connected to the decoder.
4. The method for facial posture transfer according to claim 3, characterized in that: Inputting the second mask and the missing image containing only the background into a preset inpainting network so that the inpainting network inpaints the missing image, and obtaining a complete background image of the source image comprises: The second mask and the missing image are input into an encoder in a preset inpainting network, so that the encoder extracts high-dimensional features of the missing image according to the second mask, the hierarchical fully connected layer guides the missing image to perform background repair and generation according to the high-dimensional features, and the decoder upsamples the generated missing image to restore the missing image to its original size, thereby obtaining a complete background image of the source image.
5. The method for facial posture transfer according to claim 1, characterized in that: The step of fusing the complete background image with the generated video sequence so that the complete background image replaces the background portion in the generated video sequence according to the first mask, and obtaining a posture transfer video sequence having the same background as the source image comprises: The complete background image is fused with the generated video sequence using a Poisson fusion algorithm, so that the complete background image replaces the background part in the generated video sequence according to the first mask, and a posture transfer video sequence identical to the source background is obtained.
6. A video-driven face posture migration device, characterized in that: include: An acquisition module, used to acquire source images and drive video sequences; Wherein, the source image and the driving video sequence contain human faces; An information extraction module, used to extract first facial key point information from the source image and second facial key point information from a driving video sequence; A preprocessing module, used for preprocessing the driving video sequence to obtain a facial motion track coverage area formed by facial motion in the driving video sequence; A cropping module, used for cropping the area covered by the face motion trajectory in the driving video sequence to obtain a face sub-image sequence; A feature extraction module, used to obtain a preset double-layer posture transfer network model, including: an encoder, a high-frequency feature decoder and a low-frequency feature decoder, wherein the encoder is connected to the high-frequency feature decoder and the low-frequency feature decoder, and the input end of the encoder includes a first input end and a second input end; the first facial key point information and the source image are input into the first input end of the preset double-layer posture transfer network model, and the second facial key point information and the facial sub-image sequence are input into the second input end of the double-layer posture transfer network model, so that the encoder in the double-layer posture transfer network model extracts a first feature vector of the source image based on the first facial key point information and extracts a second feature vector of the facial sub-image sequence based on the second facial key point information, the high-frequency feature decoder decodes the first feature vector to obtain a high-frequency feature map containing facial texture features in the source image; the low-frequency feature decoder decodes the second feature vector to obtain a transformation parameter representing a facial motion trend in the facial sub-image sequence and a low-frequency feature map containing contour and color features; A transformation module, used to transform the high-frequency feature map using the transformation parameters, and fuse the transformed high-frequency feature map with the low-frequency feature map to obtain a generated video sequence with a pure color background; A generating module, used for inputting the generated video sequence and the source image into a preset mask generating network respectively, to obtain a first mask of the generated video sequence and a second mask of the source image; A subtraction module, configured to use the second mask to subtract the background in the source image to obtain a missing image containing only the background; A patching module, used for inputting the second mask and the missing image into a preset patching network, so that the patching network patches the missing image to obtain a complete background image of the source image; a fusion module, configured to fuse the complete background image with the generated video sequence, so that the complete background image replaces the background part in the generated video sequence according to the first mask, and obtains a posture transfer video sequence with the same background as the source image; Wherein, the preprocessing module is specifically used for: Detecting the face position of each frame in the driving video sequence, and obtaining the face area formed by the upper left vertex and the lower right vertex in each frame; For each current frame in the driving video sequence, the IOU value of the face area of the current frame and the face track area formed by the previous frame is calculated starting from the first frame in accordance with the frame sequence. When the IOU value is greater than the threshold, it is determined that the face area of the current frame and the face track area formed by the previous frame of the current frame belong to the same target face, and the face area of the current frame is merged into the face track area formed by the previous frame of the current frame until the last frame is completed, thereby obtaining the face motion track coverage area formed by the face motion in the driving video sequence.
