A content-aware deep portrait video synthesis method

Abstract

The present invention relates to the technical field of face image and video synthesis, and discloses a content-aware deep portrait video synthesis method. The motion of the portrait area is fully modeled through a motion estimation module, and the background of the target reference image is extracted through a background estimation module, thereby obtaining sufficient full-content portrait video network input content to maximize the certainty of areas such as shoulders and background. The full-content portrait video network refines and fuses portrait conditions and background conditions, and synthesizes a video with previously generated frames. The full-content portrait video network only needs to learn to refine and fuse the content without repairing missing content, thereby ensuring certainty and stability in the synthesized video.

Description

Technical Field

[0001] The present invention relates to the technical field of synthesis of facial images and videos, and in particular to a depth portrait video synthesis method based on content perception. Background Art

[0002] As a unique attribute of identity authentication information, the human face is a research topic in numerous computer science fields. In addition to face detection, recognition, and liveness detection, the field of face synthesis is also highly active. With the recent breakthroughs in deep learning, this technology has also experienced rapid development, finding widespread application in fields such as privacy protection, film and television animation, and commercial entertainment. Compared to face detection and recognition, face synthesis is a more open field. It involves not only numerous graphics techniques such as modeling, deformation, and rendering, but also innovative image generation techniques within deep learning. While different face synthesis methods may share similar frameworks and technical approaches, their specific details and implementation methods can vary significantly. Therefore, research and exploration in face synthesis is also opening up new opportunities for the development of the technology itself. Furthermore, images or videos created using face synthesis technology are a key focus in multimedia forensics. Promoting the development of face synthesis technology will also have a positive impact on the advancement of digital forensics. Facial reenactment technology leverages the movements of a source face to manipulate a target face, forcing it to perform corresponding facial movements, enabling real-time manipulation or driving the facial expressions of animated characters. Content-aware deep portrait video synthesis methods can fully transfer the driving character's movements to the target person, synthesizing a smooth and detailed video.

[0003] Face synthesis is a key problem in deep learning, applying neural networks to face video synthesis. Conversational head reenactment aims to transfer the entire head movement from a source identity to a target identity, while maintaining full control over the target actor. It is gaining popularity in computer vision and machine learning research, and is expected to find applications in a variety of areas, including film post-production, visual dubbing, and social media. It can be categorized as either subject-agnostic or subject-specific. The difference between subject-specific and subject-independent approaches lies in whether the target actor is a specific person or a general person. This means that a trained model can only be used for video reenactment of the same person, while a trained model can be used for video reenactment of any person. Due to the limitation of using only one or a few images of the target actor, agnostic methods generally struggle to synthesize high-fidelity videos. Instead, subject-specific methods are often used to train a video of the target person. After training, a high-quality video of the person can be synthesized.

[0004] While conversational head-mounted video reenactment has achieved impressive results, generating conversational videos with good temporal coherence and generalization remains a challenging problem. Most existing methods use parameterized face models or facial landmarks to transfer motion, and then use a neural network to convert the synthetic renderings of the facial landmarks or parameterized face models into real video frames. Since the parameterized face models and facial landmarks only model the face region, the neural network input used to synthesize the final image only contains the face region. In the synthesized image, all regions except the face region are inpainted by the neural network, which has large uncertainty. As a result, the synthesized video is often unstable. In addition, for the areas that need to be inpainted, the neural network may overfit to the training set and fail to synthesize poses that do not appear in the training set, which may lead to severe artifacts.

[0005] The prior art discloses a method for generating synthetic videos based on 3D face reconstruction and video keyframe optimization. The method includes the following steps: using a convolutional neural network to optimize and fit the parameters of a 3D face morphable model to an input facial image; training a speech-to-expression and head pose mapping network using the parameters of the target video and the facial model; using the trained speech-to-expression and head pose mapping network to extract facial expression and head pose parameters from the input audio; synthesizing the face and rendering the synthesized face to generate realistic face video frames; training a rendering network based on a generative adversarial network using the parameterized face image and the facial images in the video frames, which generates a background for the face image in each frame; and performing face background rendering and video synthesis based on video keyframe optimization. In this patent, all areas of the synthesized image, except the face area, are inpainted using a neural network, resulting in significant uncertainty. As a result, the synthesized video is often unstable. Furthermore, for areas requiring inpainting, the neural network may overfit to the training set and fail to synthesize poses not present in the training set. This can cause severe artifacts, resulting in a choppy and flawed synthetic video. Summary of the Invention

[0006] The purpose of the present invention is to provide a content-aware depth portrait video synthesis method with smooth video and richer details.

[0007] To achieve the above objectives, the present invention provides a content-aware depth portrait video synthesis method, comprising:

[0008] S1. Obtain an original target video dataset, where the characters in each video of the original target video dataset are different from those in other videos;

[0009] S2. Preprocess the original target video dataset and split each video frame into a training set and a test set;

[0010] S3. Construct a portrait video synthesis model, which includes an appearance feature capture module, a motion estimation module, a background estimation module, and a full-content portrait video network. The appearance feature capture module is connected to the motion estimation module, and the motion estimation module and the background estimation module are respectively connected to the full-content portrait video network. The appearance feature capture module is used to extract appearance-related features from a given portrait image. The motion estimation module is used to predict the motion of each part of a given portrait image based on the extracted appearance features and generate portrait conditions. The background estimation module is used to capture background information and generate background conditions. The full-content portrait video network is used to fuse and refine the portrait conditions and background conditions and synthesize a video based on previously generated frames.

[0011] S4. Use the training set to train the appearance feature capture module, motion estimation module, background estimation module and full-content portrait video network respectively, and use the test set to test the effectiveness of the portrait video synthesis model to obtain a trained portrait video synthesis model;

[0012] S5. Obtain a target reference photo and a driving source video to be synthesized, where the target reference photo is a portrait of a given target actor, and the driving source video is an action video of the source actor; input the target reference photo and the driving source video into a trained target detection model, and output a driving video of the given target actor, which imitates the actions of the source actor while retaining the target actor's identity, appearance, and background changes.

[0013] As a preferred solution, the motion estimation module generates the portrait condition including:

[0014] The appearance feature capture module extracts appearance-related features from the target reference image;

[0015] The motion estimation module calculates the motion between the target reference image and the driving video frame, which includes the mapping of the movements of various parts including the face and shoulders;

[0016] The motion estimation module applies the estimated motion to the appearance-related features to obtain distorted appearance-related features, and generates a picture after the migration action based on the distorted appearance-related features.

[0017] As a preferred solution, in step S4, for the training of the motion estimation module, during training, the target reference picture and the driving video frame are from different frames of the same video, and are trained in a self-replaying manner.

[0018] As a preferred solution, in step S4, for the training of the motion estimation module, there are two mode migration actions: standard and relative.

[0019] In standard mode, the motion between the target reference image and the driving video frame is calculated and applied to the target reference image to synthesize the image after the migration action.

[0020] In the relative motion transfer mode, it is necessary to extract a reference picture with a posture similar to the target reference picture from the source video, extract the motion between the reference picture and the driving video frame and apply it to the target reference picture to obtain the picture after the transfer action.

[0021] As a preferred solution, the appearance feature capture module is an encoder, and the motion estimation module includes an unsupervised deep learning network and a decoder connected in sequence.

[0022] As a preferred solution, the background estimation module generates background conditions including:

[0023] Use image segmentation technology to process the target reference image and obtain a portrait mask covering the portrait area;

[0024] The right background image is extracted using the inverse portrait mask covering the background area;

[0025] Extract the left background image using the inverse portrait mask covering the background area;

[0026] warping the right background image according to an affine transformation between the right background image and the background of the target reference image;

[0027] Warp the left background image according to the affine transformation between it and the background of the target reference image,

[0028] The distorted right background image and the distorted left background image are fused, and the fused background image is input.

[0029] As a preferred solution, the areas of the extracted right background image and left background image are larger than half of the area of ​​the target reference image.

[0030] As a preferred solution, the background estimation module includes two encoder networks and a background fusion network. The two encoder networks are respectively connected to the background fusion network, and the two encoder networks are respectively used to process the left and right backgrounds.

[0031] As a preferred solution, in step S4, the background estimation module is trained in a self-supervised manner.

[0032] As a preferred solution, the full-content portrait video network receives three types of content as conditional inputs: portrait conditions, background conditions, and previously generated frames. These conditions are downsampled through different branches to obtain high-level features, which are then fused and upsampled into real images to generate a new frame of the video.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] The present invention fully models the motion of the portrait area through a motion estimation module, and extracts the background of the target reference image through a background estimation module, thereby obtaining sufficient full-content portrait video network input content to maximize the certainty of areas such as shoulders and background. The full-content portrait video network refines and fuses portrait conditions and background conditions, and synthesizes a video with previously generated frames. The full-content portrait video network only needs to learn to refine and fuse the content without repairing missing content, thereby ensuring certainty and stability in the synthesized video. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 4 is a flowchart of a content-aware depth portrait video synthesis method according to an embodiment of the present invention.

[0036] Figure 2 3 is a schematic diagram of a content-aware depth portrait video synthesis method according to an embodiment of the present invention.

[0037] Figure 3 2 is a schematic diagram of the extraction process of the left background and the right background in an embodiment of the present invention.

[0038] Figure 4 4 is a framework diagram of a background estimation module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0040] Example 1

[0041] like Figures 1 to 4 As shown, a content-aware depth portrait video synthesis method according to a preferred embodiment of the present invention includes:

[0042] S1. Obtain an original target video dataset, where the characters in each video of the original target video dataset are different from those in other videos;

[0043] S2. Preprocess the original target video dataset and split each video frame into a training set and a test set;

[0044] S3. Construct a portrait video synthesis model, which includes an appearance feature capture module, a motion estimation module, a background estimation module, and a full-content portrait video network. The appearance feature capture module is connected to the motion estimation module, and the motion estimation module and the background estimation module are respectively connected to the full-content portrait video network. The appearance feature capture module is used to extract appearance-related features from a given portrait image. The motion estimation module is used to predict the motion of each part of a given portrait image based on the extracted appearance features and generate portrait conditions. The background estimation module is used to capture background information and generate background conditions. The full-content portrait video network is used to fuse and refine the portrait conditions and background conditions and synthesize a video based on previously generated frames.

[0045] S4. Use the training set to train the appearance feature capture module, motion estimation module, background estimation module and full-content portrait video network respectively, and use the test set to test the effectiveness of the portrait video synthesis model to obtain a trained portrait video synthesis model;

[0046] S5. Obtain a target reference photo and a driving source video to be synthesized, where the target reference photo is a portrait of a given target actor, and the driving source video is an action video of the source actor; input the target reference photo and the driving source video into a trained target detection model, and output a driving video of the given target actor, which imitates the actions of the source actor while retaining the target actor's identity, appearance, and background changes.

[0047] This embodiment fully models the motion of the portrait area through the motion estimation module, and extracts the background of the target reference image through the background estimation module, thereby obtaining sufficient full-content portrait video network input content to maximize the certainty of areas such as shoulders and background. The full-content portrait video network refines and fuses the portrait conditions and background conditions, and synthesizes the video with the previously generated frames. The full-content portrait video network only needs to learn to refine and fuse the content without repairing the missing content, thereby ensuring certainty and stability in the synthesized video.

[0048] Specifically, the portrait image is divided into background region and portrait region, and the background region and portrait region are modeled separately. The background conditions and portrait conditions are input into the full-content portrait video network for refinement and fusion to generate new video frames.

[0049] In this embodiment, the full-content portrait video network is a neural network based on GAN (deep learning generative adversarial network).

[0050] Example 2

[0051] The difference between this embodiment and the first embodiment is that, based on the first embodiment, this embodiment further explains steps S1 and S2.

[0052] The dataset of this embodiment is the dataset released by Head2head++. This dataset consists of eight original target videos that are at least 10 minutes long. The characters in each video are different from those in other videos. The original videos are preprocessed using the same method as in Head2head++, and then the spatial resolution of each video is cropped and adjusted to 256×256 pixels. Finally, the frames of each video are split into a training set and a test set. For ease of comparison, the division of the training set and the test set is the same as that of Head2head++.

[0053] Model training for this example was performed on an NVIDIA TITAN RTX. All networks were implemented in PyTorch and optimized using Adam with default settings. Training took 60 epochs with a batch size of 5. The learning rate for the background estimation module was set to 0.00001, and for the FC2PV network to 0.00015.

[0054] The other steps of this embodiment are the same as those of the first embodiment and will not be repeated here.

[0055] Example 3

[0056] The difference between this embodiment and the second embodiment is that, based on the second embodiment, this embodiment further explains the motion estimation module and the training of the motion estimation module.

[0057] The motion estimation module generates portrait conditions including:

[0058] The appearance feature capture module extracts appearance-related features from the target reference image;

[0059] The motion estimation module calculates the motion between the target reference image and the driving video frame, which includes the mapping of the movements of various parts including the face and shoulders;

[0060] The motion estimation module applies the estimated motion to the appearance-related features to obtain distorted appearance-related features, and generates a picture after the migration action based on the distorted appearance-related features.

[0061] The appearance feature capture module of this embodiment is an encoder, and the motion estimation module includes an unsupervised deep learning network and a decoder connected in sequence.

[0062] The training of the motion estimation module is as follows:

[0063] Get the given target video and any source video , during training, a frame is selected from the source video as the target reference image , during the test, a frame is selected from the target video as the driving video frame ,and is the i-th frame of the source video (i=1,2,..,M).

[0064] First use an encoder From the target reference image Extracting appearance-related features :

[0065]

[0066] Then use the unsupervised deep learning network of the motion estimation module to calculate and driving video frames Movement between Specifically, the motion p is the driving video frame To the target reference image The mapping of the movements of various parts, including the face and shoulders, drives the motion of the video frame portrait represented by a set of key points and their local affine transformations. An optical flow is used to represent the key point information and the parameter information of the local affine transformation. These parameters enable the posture information near the key points to be obtained through the affine transformation. In addition, the unsupervised deep learning network of the motion estimation module also estimates an occlusion map The occlusion map is used to indicate the missing areas after the distortion. Then, the estimated motion is applied to In the figure, we get the distorted appearance features. :

[0067]

[0068] In the formula represents the twist operation, Represents the Hadamard product. Finally, the distorted features are passed through the decoder Generate a picture after the migration action :

[0069]

[0070] In step S4 of this embodiment, for the training of the motion estimation module, during training, the target reference image and the driving video frame are from different frames of the same video, and training is carried out in a self-replay manner. Self-replay is very useful for training and evaluating models, and can quantitatively evaluate the model's reconstruction capabilities. During the self-replay process, the source actor and the target actor are the same, achieving the effect of the character controlling himself. The training process is described by the above formula, and the image synthesized by the function constraint needs to be consistent with the driving video frame. In order to reconstruct the driving video frame from the features extracted from the reference target image, the motion estimation module must have the ability to fully model the motion of each part of the image. Therefore, after training is completed, the motion estimation module can fully model and transfer the movements of various parts, including the shoulders. This has greater advantages in motion transfer capabilities than manually annotated facial key points or some parameterized face models. After training on the portrait video dataset is completed, the model can be applied to portrait images of people with different identities.

[0071] During the test, there are two modes of migration: standard and relative. In standard mode, the calculation drives the video frame. With the target reference picture The movement between them and acts on the target reference image Above, the picture after the synthetic migration action In relative motion migration mode, it is necessary to Extract a pose and target reference image from Similar reference images , extract reference images and driving video frames The movement between them and the target reference image Go up and get the picture In this way, The action of the driver is transferred to the target character to obtain a rough target video that is consistent with the driver character's action:

[0072] 。

[0073] When extracting appearance features, this embodiment first extracts appearance-related features from the target reference image. A motion estimation module is then used to calculate the reference image and the driving video frame. The motion of the portrait in the driving video frame is represented by a set of key points and their local affine transformations. These key points are learned in an unsupervised manner, including the poses and movements of the character in areas such as the face, shoulders, neck, and upper chest. Appearance features are the person's facial expressions, such as the eyes, ears, and mouth. Generally speaking, the approach here is to preserve the expression of the target reference image and transfer the poses and movements of the driving video frame to the target reference image.

[0074] This example divides a portrait image (i.e., the target reference image) into a background region and a portrait region, and models each separately. The motion of the portrait region is represented by a set of keypoints and their local affine transformations, using an unsupervised motion model. Unlike annotated facial landmarks, these keypoints are learned in an unsupervised manner, and some may fall outside the face, such as the shoulders. Therefore, the motion of the entire upper body can be modeled and transferred from the source video to the target actor.

[0075] Furthermore, this embodiment uses FOMM (First Order Motion Model for Image Animation) to transfer the motion of the source actor to the target actor. Therefore, during cross-actor reproduction, it is first necessary to obtain a source reference image with a pose and expression similar to the target reference frame. In the experiment, a reference source frame was extracted from the source video based on the selected reference target frame. For target videos with a moving background, the input to the background estimation module can be controlled so that the background in the synthesized video is fixed or changes consistently with the original target video. Specifically, if the input target frame to the background estimation module is fixed, the background will be fixed. Conversely, if the target video sequence is fed into the background estimation module frame by frame, the background will be consistent with the input target video.

[0076] The other steps of this embodiment are the same as those of the second embodiment and will not be repeated here.

[0077] Example 4

[0078] The difference between this embodiment and the third embodiment is that, based on the third embodiment, this embodiment further explains the background estimation module and the training of the background estimation module.

[0079] The background estimation module generates background conditions including:

[0080] Use image segmentation technology to process the target reference image and obtain a portrait mask covering the portrait area;

[0081] The right background image is extracted using the inverse portrait mask covering the background area;

[0082] Extract the left background image using the inverse portrait mask covering the background area;

[0083] warping the right background image according to an affine transformation between the right background image and the background of the target reference image;

[0084] Warp the left background image according to the affine transformation between it and the background of the target reference image,

[0085] The distorted right background image and the distorted left background image are fused, and the fused background image is input.

[0086] Furthermore, the extracted right background image and left background image have an area greater than half of the target reference image area. That is, when extracting the right background image or the left background image, more than half of the area is extracted, thereby ensuring more complete background information.

[0087] like Figure 3 As shown, select the image , whose portrait is located at the far left of the target video sequence, which means that the background on the right is visible in this image. The image can be located by target detection technology or human annotation. Using image segmentation technology, a portrait mask covering the portrait area can be obtained. Then, the reverse portrait mask covering the background area is used to extract the right background image. , to avoid modeling the portrait region using the background estimation module. Finally, we Extract the correct part to obtain the correct reference background of the video sequence .

[0088] The background estimation module consists of two encoder networks ( ) and a background fusion network (H), two encoder networks ( ) are connected to the background fusion network (H), and the two encoder networks ( ) are used to process the left and right backgrounds respectively.

[0089] Reference background and the current target frame As input, where X is L or R, and predicts six real values ,…, . for and The affine background transformation between , thus:

[0090]

[0091] Then according to distortion , you can get a distorted background and As background conditions. and should match each other and with Therefore:

[0092]

[0093]

[0094] Then, a background fusion network H consisting of a series of 2D convolutional blocks is designed, which does not require any upsampling or downsampling operations to fuse and . H take and As input, output background image The size is H×W×3, where H and W are the width and height of the video frame.

[0095] 。

[0096] In this embodiment, for the background area, we assume that the background motion is rigid. During training, the background of each target frame is extracted through the background estimation module. First, the left and right reference backgrounds are extracted from the target video. The two encoder networks predict the motion between each target frame and the reference background, which is represented by the parameters of the affine transformation. The motion is then applied to the left and right reference backgrounds, and the final background of the corresponding target frame is output through the background fusion network. During testing, we reuse the background extracted from the training set as the conditional input of the neural network. Therefore, it can be based on the conditional input. Finally, the neural network is trained to refine and fuse the conditional content.

[0097] Furthermore, in step S4, the background estimation module is trained in a self-supervised manner, which avoids manual annotation.

[0098] This embodiment uses self-reproduction in both the motion estimation and background estimation modules because a ground truth frame is available. During self-reproduction, the source and target actors are identical, but the test and training sets are separate parts of the same video. Therefore, the source-driven video sequences used during testing were not seen during training. To verify the effectiveness of the background estimation module, this embodiment inputs test source frames into the background estimation module during testing for videos with moving backgrounds.

[0099] The other steps of this embodiment are the same as those of the third embodiment.

[0100] Example 5

[0101] The difference between this embodiment and the fourth embodiment is that, based on the fourth embodiment, this embodiment further illustrates the full-content portrait video network.

[0102] The Full-Content Portrait Video Network receives three types of content as conditional input: portrait conditions, background conditions, and previously generated frames. It downsamples these conditions through different branches to obtain high-level features. These high-level features are then fused and upsampled into a real image to generate a new frame of the video. This Full-Content Portrait Video Network is used for portrait video translation.

[0103] Specifically, the full-content portrait video network receives three types of content as conditional inputs, namely: portrait condition , background conditions and the previously generated frame , downsample these conditions through different branches and obtain high-level features. The obtained high-level features are fused and upsampled to the real image .therefore:

[0104]

[0105] During training, the source and target videos are the same clips. That is, in the synthesized video, each frame Should be the corresponding frame in the original target video The copy of , this frame is used as the ground truth.

[0106] During testing, the motion of the generated target in the portrait part (including the shoulders) will be based on For the background, select the background video generated by the training set as the background condition input, and also select one of the frames as the background condition input. The generated video background will be static.

[0107] The other steps of this embodiment are the same as those of the fourth embodiment and will not be repeated here.

[0108] In summary, embodiments of the present invention provide a content-aware deep portrait video synthesis method. This method uses a motion estimation module to fully model the motion of the portrait region and a background estimation module to extract the background of the target reference image. This provides sufficient input content for the full-content portrait video network, maximizing the certainty of regions such as the shoulders and background. The full-content portrait video network refines and fuses the portrait and background conditions and synthesizes the video with previously generated frames. The full-content portrait video network only needs to learn to refine and fuse the content, without repairing missing content, thus ensuring certainty and stability in the synthesized video. This method uses a recursion-free approach to synthesize a complete and realistic portrait video, avoiding cumulative errors and allowing for parallel computation. It combines the identity of the target actor with the motion of the source actor. This method fully models each component of the portrait video frame and feeds the GAN-based full conditional input into a carefully designed portrait video translation network. This network can convert the conditional input into realistic and temporally smooth video frames. The synthesized video quality of this method is superior to many recursive-based methods, demonstrating that realistic portrait videos can be synthesized without reusing previous frames.

[0109] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and substitutions can be made without departing from the technical principles of the present invention. These improvements and substitutions should also be regarded as the scope of protection of the present invention.

Claims

1. A content-aware depth portrait video synthesis method, characterized in that: include: S1. Obtain an original target video dataset, where the characters in each video of the original target video dataset are different from those in other videos; S2. Preprocess the original target video dataset and split each video frame into a training set and a test set; S3. Construct a portrait video synthesis model, which includes an appearance feature capture module, a motion estimation module, a background estimation module, and a full-content portrait video network. The appearance feature capture module is connected to the motion estimation module, and the motion estimation module and the background estimation module are respectively connected to the full-content portrait video network. The appearance feature capture module is used to extract appearance-related features from a given portrait image; The motion estimation module is used to predict the motion of each part of a given portrait image based on the extracted appearance features and generate portrait conditions; the background estimation module is used to capture background information and generate background conditions; The full-content portrait video network is used to fuse and refine portrait conditions and background conditions, and synthesize videos based on previously generated frames; The appearance feature capture module is an encoder, and the motion estimation module includes an unsupervised deep learning network and a decoder connected in sequence; The training of the motion estimation module is as follows: Get the given target video and any source video , during training, a frame is selected from the source video as the target reference image , during the test, a frame is selected from the target video as the driving video frame ,and is the i-th frame of the source video, i=1,2,..,M; First use an encoder From the target reference image Extract appearance-related features : Then use the unsupervised deep learning network of the motion estimation module to calculate and driving video frames Movement between Specifically, the motion p is the driving video frame To the target reference image The mapping of the movements of various parts, including the face and shoulders, drives the motion of the video frame portrait represented by a set of key points and their local affine transformations. An optical flow is used to represent the key point information and the parameter information of the local affine transformation. These parameters enable the posture information near the key points to be obtained through the affine transformation. In addition, the unsupervised deep learning network of the motion estimation module also estimates an occlusion map. ; The occlusion map is used to indicate the missing areas after the distortion; then, the estimated motion is applied to In the figure, we get the distorted appearance features. : In the formula represents the twist operation, Represents the Hadamard product; finally, the distorted features are passed through the decoder Generate a picture after the migration action : ； S4. Use the training set to train the appearance feature capture module, motion estimation module, background estimation module and full-content portrait video network respectively, and use the test set to test the effectiveness of the portrait video synthesis model to obtain a trained portrait video synthesis model; S5. Obtain a target reference photo and a driving source video to be synthesized, where the target reference photo is a portrait of a given target actor, and the driving source video is an action video of the source actor; The target reference photo and the driving source video are fed into the trained object detection model, which outputs a driving video of the given target actor that imitates the movements of the source actor while preserving the target actor's identity, appearance, and background changes.

2. The content-aware depth portrait video synthesis method according to claim 1, characterized in that: The motion estimation module generates portrait conditions including: The appearance feature capture module extracts appearance-related features from the target reference image; The motion estimation module calculates the motion between the target reference image and the driving video frame, which includes the mapping of the movements of various parts including the face and shoulders; The motion estimation module applies the estimated motion to the appearance-related features to obtain distorted appearance-related features, and generates a picture after the migration action based on the distorted appearance-related features.

3. The content-aware depth portrait video synthesis method according to claim 2, characterized in that: In step S4, for the training of the motion estimation module, during training, the target reference picture and the driving video frame are from different frames of the same video, and are trained in a self-replaying manner.

4. The content-aware depth portrait video synthesis method according to claim 3, characterized in that: In step S4, for the training of the motion estimation module, there are two modes of migration actions: standard and relative. In standard mode, the motion between the target reference image and the driving video frame is calculated and applied to the target reference image to synthesize the image after the migration action. In the relative motion transfer mode, it is necessary to extract a reference picture with a posture similar to the target reference picture from the source video, extract the motion between the reference picture and the driving video frame and apply it to the target reference picture to obtain the picture after the transfer action.

5. The content-aware depth portrait video synthesis method according to claim 1, characterized in that: The appearance feature capture module is an encoder, and the motion estimation module includes an unsupervised deep learning network and a decoder connected in sequence.

6. The content-aware depth portrait video synthesis method according to claim 1, characterized in that: The background estimation module generates background conditions including: Use image segmentation technology to process the target reference image and obtain a portrait mask covering the portrait area; The right background image is extracted using the inverse portrait mask covering the background area; Extract the left background image using the inverse portrait mask covering the background area; warping the right background image according to an affine transformation between the right background image and the background of the target reference image; Warp the left background image according to the affine transformation between it and the background of the target reference image, The distorted right background image and the distorted left background image are fused, and the fused background image is input.

7. The content-aware depth portrait video synthesis method according to claim 6, characterized in that: The extracted right background image and left background image have an area greater than half of the area of ​​the target reference image.

8. The content-aware depth portrait video synthesis method according to claim 6, characterized in that: The background estimation module includes two encoder networks and a background fusion network. The two encoder networks are connected to the background fusion network respectively, and the two encoder networks are used to process the left and right backgrounds respectively.

9. The content-aware depth portrait video synthesis method according to claim 1, characterized in that: In step S4, the background estimation module is trained in a self-supervised manner.

10. The content-aware depth portrait video synthesis method according to claim 1, characterized in that: The full-content portrait video network receives three types of content as conditional inputs: portrait conditions, background conditions, and previously generated frames. It downsamples these conditions through different branches and obtains high-level features. The obtained high-level features are fused and upsampled into real images to generate a new frame of the video.

Images