A method and system for zero-shot video-to-video expression transfer

Abstract

The application provides a method and system for zero-shot video-to-video expression transfer, which comprises the following steps: obtaining a source video and an expression video, wherein a face in the source video is a to-be-transferred object of a face in the expression video; preprocessing the source video and the expression video to obtain a target source video and a target expression video; performing video expression transfer according to a target face global feature, a target motion feature and a target expression transfer model to obtain a transferred video; the target face global feature is obtained by calculating the target source video through the target expression transfer model; the target motion feature is obtained by calculating the target expression video through the target expression transfer model; the video expression transfer comprises face feature extraction, face feature global analysis, face motion analysis, face volume deformation and rendering processing. The above-mentioned scheme solves the problem of poor expression transfer effect of the existing video transfer technology.

Description

Technical Field

[0001] This invention relates to the field of video expression transfer technology, and in particular to a method and system for zero-sample video to video expression transfer. Background Technology

[0002] Facial expression transfer is a crucial area of ​​computer vision and graphics, widely applied in virtual reality, filmmaking, game development, and human-computer interaction. The core of facial expression transfer technology lies in transferring the expression features of a source face to a target face, enabling the target face to generate the same dynamic facial expressions as the source face. Typically, this technology relies on feature extraction from the source face, combined with deep learning or traditional graphics methods, to generate the corresponding expression changes on the target face. By using expression transfer technology, real-time synchronization of expressions from different faces can be achieved, thereby improving the realism of virtual characters and the interactive experience.

[0003] Video-to-video facial expression transfer technology faces a series of technical challenges, mainly including the following aspects:

[0004] Accuracy of Facial Feature Extraction: To achieve high-quality facial expression transfer, it is essential to accurately extract the facial expression features of the source face. However, due to the complexity of the face, varying lighting conditions, and occlusion issues, the extracted facial expression features may be affected, leading to unsatisfactory transfer results.

[0005] Cross-facial diversity processing: Different individuals have significant differences in facial structure, such as facial proportions, skin color, gender, and age, which poses additional challenges to expression transfer. How to accurately and naturally map expressions between different faces is a major technical challenge.

[0006] Temporal consistency and coherence: For dynamic videos or real-time interactions, facial expression transfer not only needs to handle static expressions in each frame, but also needs to ensure temporal consistency during the transfer process to avoid jumps or unnatural transitions in facial expressions between consecutive frames. Especially during the transfer between videos, it is necessary to ensure that only facial expressions are changed without affecting the head movement in the source video, while preventing unnatural issues such as jitter.

[0007] Micro-expression capture: In addition to large facial expressions, the delicate capture and transmission of micro-expressions is also an important factor in improving the realism of facial transfer. Current technology still faces difficulties in processing subtle and transient changes in expression, especially in low-resolution video or under less than ideal shooting conditions.

[0008] Preservation of detail and texture: Maintaining the facial details and skin texture of the target face during expression transfer, without distortion or interference from the source face, is another key challenge. This is especially true under significant changes in expression, which can easily lead to distortion or blurring of facial details. Summary of the Invention

[0009] This application provides a method and system for zero-sample video-to-video facial expression transfer to solve the problem of poor facial expression transfer effect in existing video transfer technologies.

[0010] Firstly, this application provides a method for zero-shot video-to-video facial expression transfer, the method comprising:

[0011] Obtain the source video and the expression video, wherein the face in the source video is the object to be migrated from the face in the expression video;

[0012] The source video and the expression video are preprocessed to obtain the target source video and the target expression video;

[0013] Video expression transfer is performed based on the target face global features, target motion features, and target expression transfer model to obtain a transferred video; the target face global features are obtained by calculating the target source video using the target expression transfer model; the target motion features are obtained by calculating the target expression video using the target expression transfer model; the video expression transfer includes global face feature analysis, face motion analysis, face volume deformation, and rendering processing.

[0014] Preferably, the target facial expression transfer model includes:

[0015] A facial expression motion estimation module is configured to perform facial motion analysis based on the target expression video and the target source video to obtain the global features of the target face and the target motion features.

[0016] Preferably, the facial expression motion estimation module includes:

[0017] The first feature coding unit and the second feature coding unit, wherein there are multiple first feature coding units, and the multiple first feature coding units are stacked in a layered distribution;

[0018] The first feature encoding unit includes:

[0019] A feature grouping subunit is configured to divide an input image sequence into multiple first sequences; the image sequences include image sequences obtained based on the target expression video, and / or image sequences output by the previous level's first feature encoding unit;

[0020] The feature processing subunit includes multiple feature processing layers distributed in parallel. The multiple feature processing layers are configured to perform feature encoding processing on multiple first sequences respectively, so as to output corresponding second sequences respectively.

[0021] A feature fusion subunit is configured to perform feature fusion based on multiple second sequences to obtain first image features;

[0022] The second feature encoding unit is configured to perform feature encoding processing based on the first image features output by each of the first feature encoding units to obtain the second image features;

[0023] The facial expression motion estimation module is further configured to obtain the target motion features or the target face global features based on the first image features and the second image features.

[0024] Preferably, the feature processing subunit further includes multiple attention fusion layers, and the multiple attention fusion layers correspond one-to-one with the multiple feature processing layers;

[0025] The attention fusion layer is configured to: calculate attention weights based on the first sequence and the second sequence output by another feature processing subunit, and perform feature fusion processing on the first sequence and the second sequence based on the attention weights;

[0026] The feature processing layer is further configured to perform feature encoding processing based on the fusion result of the first sequence and the second sequence to output the second sequence.

[0027] Preferably, the target expression transfer model further includes:

[0028] A feature fusion module is configured to perform feature fusion based on the global features of the target face and the target motion features to obtain target fused features;

[0029] A coordinate module is configured to import the target fusion feature into a regularized coordinate system and perform face volume deformation on the target fusion feature in the regularized coordinate system to obtain target migration features;

[0030] A rendering module is configured to perform rendering processing based on the target migration features to obtain the target migration video.

[0031] Preferably, the coordinate module is further configured as follows:

[0032] In the regularized coordinate system, the volumetric deformation of the fused feature is performed using the head rotation angle corresponding to the first training video, and the size of the transferred feature is the same as that of the face feature.

[0033] Preferably, the training process of the target facial expression transfer model includes:

[0034] Acquire training videos, wherein the training videos include at least a first training video and a second training video;

[0035] The first training video and the second training video are preprocessed respectively to obtain the first video and the second video;

[0036] The target expression transfer model is obtained by training the model based on the first video, the second video, and the expression transfer model.

[0037] Preferably, the step of preprocessing the first training video and the second training video respectively includes:

[0038] The first training video is sequentially subjected to initial feature extraction, image cropping, and image organization and filtering to obtain the first video.

[0039] The second training video is cropped to obtain the second video.

[0040] Preferably, the initial feature extraction includes performing face position detection, facial key point extraction, and head rotation angle extraction sequentially based on the first training video to obtain the face image and face data; the face data includes several facial key points and head rotation angle.

[0041] The image cropping includes aligning the center point of the face in the face image with the center point of the face image based on the facial key points, and performing uniform cropping without deformation to obtain a cropped image.

[0042] The image organization and filtering includes filtering based on the cropped image to obtain a first video, the first video including several frames of a human face in a natural state.

[0043] Preferably, the step of training the model based on the first video, the second video, and the expression transfer model includes:

[0044] Facial feature analysis is performed based on the first video to obtain global facial features;

[0045] Facial motion analysis is performed on the second video to obtain motion features;

[0046] The motion features and the global facial features are fused to obtain the fused features;

[0047] Based on the fusion features, volume deformation is performed to obtain migration features, and the migration features are rendered to obtain a migration video;

[0048] Repeat the training until the transfer video meets the requirements of the preset loss function to obtain the target expression transfer model.

[0049] Preferably, the step of performing volumetric deformation and rendering based on the fusion features includes:

[0050] The transfer features are rendered and a non-binary mask is set to limit the rendering process to obtain a transfer video. This process continues until the transfer video meets the requirements of the loss function, thus obtaining the target expression transfer model.

[0051] Secondly, this application also provides a system for zero-sample video-to-video facial expression transfer, the system being used to perform the above-described method, the system comprising:

[0052] The data acquisition module is configured to acquire source video and expression video, wherein the face in the source video is the face to be migrated in the expression video;

[0053] A preprocessing module is configured to preprocess the source video and the expression video to obtain a target source video and a target expression video.

[0054] A target expression transfer module is constructed based on a target expression transfer model. This module is configured to perform video expression transfer based on global features of the target face, target motion features, and the target expression transfer model to obtain a transferred video. The global features of the target face are calculated from the target source video using the target expression transfer model. The target motion features are calculated from the target expression video using the target expression transfer model. The video expression transfer includes global facial feature analysis, facial motion analysis, facial volume deformation, and rendering processing.

[0055] As described above, this application provides a method and system for zero-shot video-to-video expression transfer. The method includes acquiring a source video and an expression video, wherein the face in the source video is the object to be transferred from the face in the expression video; preprocessing the source video and the expression video to obtain a target source video and a target expression video; performing video expression transfer based on the target face global features, target motion features, and a target expression transfer model to obtain a transferred video; the target face global features are calculated from the target source video using the target expression transfer model; the target motion features are calculated from the target expression video using the target expression transfer model; the video expression transfer includes face feature extraction, global face feature analysis, face motion analysis, face volume deformation, and rendering processing. This application solves the problem of poor expression transfer effect in existing video transfer technologies through the above solution. Attached Figure Description

[0056] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 This is a flowchart illustrating a zero-shot video-to-video facial expression transfer method according to this application;

[0058] Figure 2 This is a schematic diagram of the target expression transfer model described in the zero-shot video-to-video expression transfer method of this application;

[0059] Figure 3 This is a flowchart illustrating the training process of the target expression transfer model in the zero-shot video-to-video expression transfer method of this application.

[0060] Figure 4 This is a flowchart illustrating the video preprocessing training in a zero-shot video-to-video facial expression transfer method according to this application;

[0061] Figure 5 This is a flowchart illustrating the specific training process of the expression transfer model in the zero-shot video-to-video expression transfer method of this application.

[0062] Figure 6 This is a diagram showing the relationship between the target expression transfer model and the processed data in the zero-shot video-to-video expression transfer method of this application;

[0063] Figure 7 This is a schematic diagram of a zero-sample video-to-video facial expression transfer system according to this application;

[0064] Figure 8 This is a schematic diagram of the facial expression motion estimation module in a zero-shot video-to-video expression transfer method of this application;

[0065] Figure 9 This is a schematic diagram of the first feature encoding unit in a zero-sample video-to-video expression transfer method of this application. Detailed Implementation

[0066] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] Figure 1This is a flowchart illustrating a zero-sample video-to-video facial expression transfer method according to this application.

[0068] See Figure 1 As can be seen, this embodiment provides a method for zero-sample video-to-video facial expression transfer, the method including:

[0069] S100, Obtain the source video and the expression video. Specifically, in this embodiment, before performing video expression migration, it is necessary to first obtain the expression migration video and the expression providing video. The source video is the expression migration video, and the expression video is the expression providing video. Therefore, it can be understood that the face in the source video is the object to be migrated from the face in the expression video.

[0070] It should be noted that this embodiment can complete video expression transfer by using expression transfer video and expression providing video, without the need to adapt the face beforehand during the video expression transfer process. Therefore, it can achieve video expression transfer with zero sample videos.

[0071] The method further includes:

[0072] S200, preprocess the source video and the expression video to obtain the target source video and the target expression video. Specifically, in this embodiment, the source video and the expression video need to be preprocessed before the video expression migration is performed. Since the obtained source video and / or expression video may have problems such as non-standard face position, excessive facial expression amplitude, and low video clarity, they need to be preprocessed before the subsequent video expression migration process can be carried out.

[0073] The method further includes:

[0074] S300, video expression transfer is performed based on the global features of the target face, the target motion features, and the target expression transfer model to obtain the transferred video. Specifically, in this embodiment, the video expression transfer is performed on the global features of the target face and the target motion features using the pre-trained target expression transfer model. During the expression transfer process, the transferred face has already been combined with the source video, so there is no need to perform face transfer again after the expression transfer.

[0075] This embodiment achieves facial expression transfer by combining facial features from the source video as a frame reference and incorporating facial motion prediction from the target video. Specifically, the global features of the target face are calculated from the target source video using the target expression transfer model; the target motion features are calculated from the target expression video using the target expression transfer model; and the video expression transfer includes facial feature extraction, global facial feature analysis, facial motion analysis, facial volume deformation, and rendering processing.

[0076] It should be noted that the preprocessing is also completed in the target expression transfer model, so there is no need to preprocess the acquired source video and the expression video, thereby improving the efficiency of expression transfer.

[0077] Figure 2 This is a schematic diagram of the target expression transfer model described in the zero-sample video-to-video expression transfer method of this application.

[0078] See Figure 2 Furthermore, in some embodiments, the target facial expression transfer model includes:

[0079] A facial expression motion estimation module is configured to perform facial motion analysis based on the target expression video to obtain the target motion features and the target source video. Specifically, in this embodiment, the facial expression motion estimation module performs facial estimation on the face in the target expression video or the target source video to obtain facial expressions that are closer to real human faces. Combined with facial features, this enables the transferred face to achieve smooth expression switching.

[0080] Figure 8 This is a schematic diagram of the facial expression motion estimation module in a zero-sample video-to-video expression transfer method of this application.

[0081] See Figure 8 Furthermore, in some embodiments, the facial expression motion estimation module includes:

[0082] The first feature coding unit and the second feature coding unit, wherein there are multiple first feature coding units, and the multiple first feature coding units are stacked in a layered distribution;

[0083] The first feature encoding unit includes:

[0084] A feature grouping subunit is configured to divide an input image sequence into multiple first sequences; the image sequences include image sequences obtained based on the target expression video, and / or image sequences output by the previous level's first feature encoding unit;

[0085] The feature processing subunit includes multiple feature processing layers distributed in parallel. The multiple feature processing layers are configured to perform feature encoding processing on multiple first sequences respectively, so as to output corresponding second sequences respectively.

[0086] A feature fusion subunit is configured to perform feature fusion based on multiple second sequences to obtain first image features;

[0087] The second feature encoding unit is configured to perform feature encoding processing based on the first image features output by each of the first feature encoding units to obtain the second image features;

[0088] The facial expression motion estimation module is further configured to obtain the target motion features or the target face global features based on the first image features and the second image features.

[0089] Specifically, in this embodiment, the facial expression motion estimation module completes facial analysis of the target source video or the target expression video through the first feature encoding unit and the second feature encoding unit.

[0090] Specifically, the first feature encoding unit performs corresponding optimization processing on local features in the video processing process, and the second feature encoding unit performs corresponding optimization processing on global features in the image processing process; the analysis and extraction of local and global features are achieved through the first feature encoding unit and the second feature encoding unit, thereby further improving the extraction effect of the target motion features or the target face global features.

[0091] Specifically, the first feature encoding unit uses a feature processing subunit with multiple feature processing layers to encode features in different image sequences and then fuses the encoded sequences to extract local features; the second feature encoding unit further extracts features from the sequence output by the first feature encoding unit to obtain global features.

[0092] It should be noted that, since the facial expression motion estimation module has the function of extracting features from both the target source video and the target expression video, the processing method is slightly different for different video sources. For the target expression video that provides the expression, the facial expression motion estimation module needs to extract features that are more inclined towards static facial features based on the target expression video. For the target source video that carries the expression, the facial expression motion estimation module needs to extract features that are more inclined towards facial movement based on the target source video.

[0093] Figure 9 This is a schematic diagram of the first feature encoding unit in a zero-sample video-to-video expression transfer method of this application.

[0094] See Figure 9 It can be seen that, further, in some embodiments, the feature processing subunit further includes multiple attention fusion layers, and the multiple attention fusion layers correspond one-to-one with the multiple feature processing layers;

[0095] The attention fusion layer is configured to: calculate attention weights based on the first sequence and the second sequence output by another feature processing subunit, and perform feature fusion processing on the first sequence and the second sequence based on the attention weights;

[0096] The feature processing layer is further configured to perform feature encoding processing based on the fusion result of the first sequence and the second sequence to output the second sequence.

[0097] Specifically, in this embodiment, considering that the local features extracted by the first feature encoding unit may still have some subtle features that cannot be accurately extracted, this embodiment sets the attention fusion layer in the first feature encoding unit. By having multiple attention fusion layers cooperate with the feature processing layer, a more accurate technical foundation can be provided for the extraction of local features.

[0098] Specifically, by adding an attention weight mechanism to the sequence output by the feature processing subunit, the distribution of features in the sequence is made more consistent with actual human facial features, thereby further improving the realism of the transferred face and avoiding any unnaturalness in the generated face. Furthermore, in some embodiments, the target expression transfer model also includes:

[0099] A feature fusion module is configured to fuse the global features of the target face and the target motion features to obtain target fused features. Specifically, in this embodiment, since faces are relatively complex and facial expressions frequently change during facial expression transfer, and expression changes cannot be smoothly achieved through a single feature or motion prediction, this embodiment utilizes the feature fusion module to fuse the global features of the target face and the target motion features to obtain the target fused features that simultaneously possess the facial features from the source video and the facial expressions from the expression video. This can be understood as the feature fusion module combining key elements from both videos to improve the video quality after facial expression transfer.

[0100] The target facial expression transfer model also includes:

[0101] The coordinate module is configured to import the target fusion feature into a regularized coordinate system and perform facial volume deformation on the target fusion feature in the regularized coordinate system to obtain the target transfer feature. Specifically, in this embodiment, since the parameters of the faces in different videos are different, it is necessary to deform the fused target fusion feature to a certain extent. Placing the feature in a specific coordinate system for deformation is beneficial to the completion of the deformation. Therefore, in this embodiment, the coordinate module imports the target fusion feature into a regularized coordinate system and performs facial volume deformation on the target fusion feature in the regularized coordinate system to obtain the target transfer feature that conforms to the face size after expression transfer.

[0102] It should be noted that since a human face is not a single 2D image but a 3D structure, attention should be paid to the deformation of the face volume during the deformation process.

[0103] It should be noted that, in the regularized coordinate system, the volumetric deformation of the fused feature is performed using the head rotation angle corresponding to the first training video, and the size of the transferred feature is the same as that of the face feature.

[0104] The target facial expression transfer model also includes:

[0105] A rendering module is configured to perform rendering processing based on the target migration features to obtain a target migration video. Specifically, in this embodiment, after obtaining the target migration features, a single target migration feature cannot be directly used as the migrated expression. It needs to be rendered based on the target migration features before a corresponding video can be generated. Therefore, this embodiment introduces the rendering module to perform rendering processing on the target migration features to obtain the target migration video after expression migration.

[0106] Figure 3 This is a flowchart illustrating the training process of the target expression transfer model in the zero-shot video-to-video expression transfer method of this application.

[0107] See Figure 3 It is understood that, furthermore, in some embodiments, the training process of the target expression transfer model includes:

[0108] S010, Obtain training videos. The training videos include at least a first training video and a second training video. Specifically, in this embodiment, similar to step S100, before training the expression transfer model, it is also necessary to obtain video samples for training. The obtained video samples can be divided into two categories: one is a video used to provide expressions, and the other is a video waiting to transfer expressions. In this embodiment, the first training video is a video waiting to transfer expressions, and the second training video is a video providing expressions.

[0109] The training process of the target facial expression transfer model also includes:

[0110] S020, the first training video and the second training video are preprocessed respectively to obtain the first video and the second video. Specifically, in this embodiment, the preprocessing method is different for different videos.

[0111] Figure 4 This is a flowchart of the video preprocessing process used in the zero-shot video-to-video facial expression transfer method of this application.

[0112] See Figure 4 It can be seen that the steps of preprocessing the first training video and the second training video respectively include:

[0113] S021, perform initial feature extraction, image cropping and image organization filtering on the first training video in sequence to obtain the first video;

[0114] S022, perform image cropping on the second training video to obtain the second video.

[0115] Specifically, in this embodiment, for the first training video, initial feature extraction, image cropping, and image organization filtering need to be performed sequentially to obtain the first video; for the second training video, only image cropping needs to be performed to obtain the second video.

[0116] The initial feature extraction includes sequentially performing face position detection, facial key point extraction, and head rotation angle extraction based on the first training video to obtain the face image and face data; the face data includes several facial key points and head rotation angle; the image cropping includes aligning the center point of the face in the face image with the center point of the face image based on the facial key points, and performing uniform cropping without deformation to obtain a cropped image; the image organization and filtering includes filtering based on the cropped image to obtain a first video, the first video including several frames of the face in a natural state.

[0117] It should be noted that during the formal expression transfer, the preprocessing of the target video is the same as that of the second training video, and the preprocessing of the source video is the same as that of the first training video.

[0118] S030, the target expression transfer model is obtained by training the model based on the first video, the second video, and the expression transfer model. Specifically, in this embodiment, the first video and the second video are imported into the expression transfer model for model training, wherein the expression transfer model is the target expression transfer model that has not been trained or has not completed training.

[0119] It should be noted that the training of the expression transfer model cannot be completed by using only the first training video and the second training video. It requires training several times using several videos that are the same as the first training video and the second training video to obtain the target expression transfer model.

[0120] Figure 5 This is a flowchart illustrating the specific training process of the expression transfer model in the zero-shot video-to-video expression transfer method of this application.

[0121] See Figure 5 Therefore, in this embodiment, the step of training the model based on the first video, the second video, and the expression transfer model includes:

[0122] S031, Perform facial feature analysis based on the first video to obtain global facial features;

[0123] S032, Perform facial motion analysis based on the second video to obtain motion features;

[0124] S033, the motion features and the global facial features are fused to obtain fused features;

[0125] S034, Perform volume deformation based on the fusion features to obtain migration features, and render the migration features to obtain a migration video;

[0126] S035, repeat the training until the transfer video meets the requirements of the preset loss function to obtain the target expression transfer model.

[0127] Specifically, in this embodiment, the model training steps S031 to S035 are similar to step S300. Both involve first extracting facial features from the video of the expression to be transferred; performing global analysis on the facial features to obtain global facial features; predicting facial motion in the video of the expression to obtain motion features; and combining the global facial features and motion features and rendering them to obtain the video after expression transfer. The difference is that the model training process requires multiple expression transfers to obtain the final target expression transfer model.

[0128] In the process of volume deformation and rendering based on the fusion features, it is necessary to render the transfer features and set a non-binary mask to limit the rendering process, so as to obtain the transfer video until the transfer video meets the requirements of the loss function, thereby obtaining the target expression transfer model.

[0129] It should be noted that acquiring the source video and the expression video, preprocessing the source video and the expression video respectively, acquiring the first training video and the second training video and preprocessing the first training video and the second training video respectively all have one thing in common: the acquisition or preprocessing of the two types of videos is performed simultaneously.

[0130] Figure 7 This is a schematic diagram of a system for zero-sample video-to-video facial expression transfer according to this application.

[0131] See Figure 7 It is understood that this embodiment also provides a system for zero-sample video-to-video facial expression transfer, the system being used to execute the above method, the system comprising:

[0132] The data acquisition module 10 is configured to acquire source video and expression video. The face in the source video is the object to be transferred from the face in the expression video. Specifically, in this embodiment, the data acquisition module 10 is used to acquire the source video and the expression video. However, it should be noted that the source video and the expression video are both video sources in the actual expression transfer. During the model training process, the data acquisition module 10 can also acquire video sources used for training.

[0133] The system also includes:

[0134] The preprocessing module 20 is configured to preprocess the source video and the expression video to obtain a target source video and a target expression video. Specifically, in this embodiment, the preprocessing module 20 is used to preprocess the source video and the expression video. It should be noted that the types of preprocessing performed on the source video and the expression video are different, and depending on the actual video conditions of the source video and the expression video, preprocessing may not be performed. Similar to the data acquisition module 10 described above, the preprocessing module 20 is not only used in the practical application of expression transfer, but can also be used in the process of model training.

[0135] The system also includes:

[0136] The target expression transfer module 30 is constructed based on a target expression transfer model. The target expression transfer module 30 is configured to perform video expression transfer based on global features of the target face, target motion features, and the target expression transfer model to obtain a transferred video. The global features of the target face are calculated from the target source video using the target expression transfer model. The target motion features are calculated from the target expression video using the target expression transfer model. The video expression transfer includes global facial feature analysis, facial motion analysis, facial volume deformation, and rendering processing.

[0137] Specifically, in this embodiment, the target expression transfer module 30 extracts features from the target source video and the target expression video respectively, and merges the two features together to achieve expression transfer; wherein, the target expression transfer module 30 simultaneously extracts local features and global features of the image and merges the two to improve the realism of the face after expression transfer.

[0138] Figure 6 This diagram illustrates the relationship between the target expression transfer model and the processed data in the zero-shot video-to-video expression transfer method of this application.

[0139] See Figure 6 As can be seen, this embodiment also provides two specific implementation methods: model training and video expression transfer.

[0140] The principle of the solution in this embodiment is as follows:

[0141] First, estimate and extract the facial features (f) of the face in the current frame of the source video in its natural state. a Estimate the facial pose of the source video, and estimate the facial motion features of the target video (f) by performing facial motion estimation on the target video. e ), combined with the face pose of the source video (f p ) and target video facial expression motion features (f e ) for fa Deformation is performed to obtain the deformed facial features. The face rendering and fusion module is used to generate the migrated face and render it onto the face area in the source video frame.

[0142] Example 1 (Model Training):

[0143] S1.1 Training Data Preprocessing mainly consists of four parts: data acquisition, feature extraction, data cropping, and data organization and filtering. Data acquisition uses any high-definition shooting equipment, ensuring clear facial images. The captured content should include segments of people speaking, from multiple angles, and with rich facial expressions. Feature extraction mainly includes face position detection, facial key point extraction, and estimation of head rotation angles. Data cropping is a process of uniformly cropping without deformation by aligning the face center point with the image center position frame by frame based on key points. Data organization and filtering aims to select frames with natural facial expressions and balance the distribution of various head movements and expression categories. This process mainly combines the distribution of facial key points, the distance ratio coefficients of relevant key points in the eye and mouth regions, and facial expression classification to classify and organize the facial attributes of each video frame, then balances the results, and finally filters out frames with natural facial expressions separately. It should be noted that in the context of this invention, we define a natural facial expression frame as follows: the detected head yaw angle is in the range (-10°, 10°), the roll angle is in the range (-8°, 8°), the pitch angle is in the range (-10°, 10°), the eye opening / closing ratio is approximately (0.3, 0.5), and the mouth opening / closing ratio is (5e... -3 ,2e -2 The expression is classified as neutral with a confidence level of over 80%, while the confidence level of the other categories is less than 45%.

[0144] S1.2 Pre-trained facial feature estimation and extraction module for natural facial expressions uses residual convolutional network blocks (ResBlock) as the basic module to construct an autoencoder (E... A D A This algorithm takes any frame from any video segment as input and outputs a target (groundtruth, gt) that corresponds to any natural facial expression frame in the video. In this process, we define the following objective function:

[0145]

[0146] in These represent the input frame features obtained by the encoder part of the video frame after passing through the autoencoder, and the features of their corresponding natural facial expression frames, respectively. For the reconstruction of the decoder output of the last input frame, I neuThe input consists of natural facial expression frames. It's important to note that the features here are 3D volumetric features. The decoder within this autoencoder is constructed using a decoder network with projection properties, and this decoder is a coarse decoder used only to aid training.

[0147] S1.3 End-to-end training of facial expression motion estimation, motion deformation construction, and facial rendering fusion. The facial expression motion estimation module is a network structure based on an improved version of an existing CNN. Through this network, we obtain the latent space feature representation z of facial expression motion. m Regarding the 3D facial features mentioned in S1.2, a global facial feature z is extracted in this stage using a branch module based on an improvement of existing CNNs. id ; will z m With z id The fusion yields a motion semantic representation corresponding to the current character's facial ID. In the motion deformation construction stage, a 1x1 convolutional structure is used to represent the global motion. Unfold into a 2x2x2 volume representation with spatial local structure, and extend this representation to f using multi-level 3D residual convolutional blocks. a The volume representation (of the natural appearance features) is scale-consistent. This representation is then used to calculate a volume motion estimate relative to a regularized coordinate system through an additionally defined prediction layer and offset. Specifically, this volume motion estimate can be understood as an implicit 3D optical flow representation corresponding to the regularized coordinate system. Facial expression movement The head rotation angle R of the detected input frame is used to sequentially apply volumetric deformation to the standardized facial representation f of the input frame. a Thus, we obtain z with facial expression movement. m Complete facial features with head movement angle R Using D a Doubling the number of network layers to build the rendering fusion module G enables The network outputs the complete image of the current input frame after expression transfer. In this stage S1.2, E... A The input and reconstruction target are two different frames in the same video segment k, which will be frozen and not used in training. Specifically, the above f a Should come from Motion-related features and head motion response extracted from In this stage, we use the following objective function to constrain the entire network:

[0148]

[0149] in express Migrate to The reconstruction losses, To combat the losses, To reconstruct the ID feature constraints of the face, This is a reuse of the objective function in S1.2, used to control the distribution consistency of the facial representation of standard faces. λ1 and λ2 are adjustable hyperparameters. This process enables texture-consistent migration motion effects while ensuring sufficiently rich facial expression details.

[0150] S1.4 Refines facial rendering fusion, ensuring seamless integration of the current frame into the complete video frame image after facial expression transfer is completed. Building upon the successful training in S1.3, an additional motion adaptation layer M is introduced into the facial expression motion estimation layer. adp At this stage, only M adp Train the algorithm and fine-tune G. Use heterogeneous ID video clips as input. As the source provides the facial representation f a With head movement R, As an emoji target, z is provided m After a migration process similar to S1.3, but it makes Under condition R, the motion adaptation adjustment layer M is passed once. adp The remaining process is consistent with that described in S1.3, completing one expression transfer to obtain the output. We define an appropriate non-binary mask for... Regional constraints are used to reconstruct the outer edges. Internal reconstruction without going through M adp Time-transfer output We define the objective function for this stage as follows:

[0151]

[0152] in This indicates the reconstruction of the outer area of ​​the mask. This indicates the reconstruction of the internal area of ​​the mask. This indicates self-reconstruction consistent with S1.3. For the continuity constraint of the motion representation, λ c This allows for adjustable hyperparameters. This stage-based approach can address various issues related to inter-video migration, such as patching and maintaining the continuity of migration motion.

[0153] Non-binary masks, as an image processing technique, are commonly used to control the display and hiding of specific areas within layers or video frames. The grayscale value range of a non-binary mask is from 0 (completely transparent) to 255 (completely opaque), where gray areas represent different degrees of semi-transparency. This type of mask has wide applications in film and television editing and image processing. For example, non-binary masks can be used to achieve smooth transitions between video clips or to create various complex overlay and gradient effects. During video generation, non-binary masks help creators more precisely control the displayed areas of the video frame, resulting in more natural and fluid visual effects.

[0154] Example 2 (Video Expression Transfer):

[0155] S2.1 Input Video Preprocessing: Given a source video segment First, the source video undergoes preprocessing, including face region cropping and center alignment, to extract head rotation parameters for each frame. Then, a target expression video segment is selected. The facial region of the target expression video is cropped using a fixed bounding box coordinate cropping method.

[0156] S2.2 Model Inference: Input to encoder E a Encode to obtain facial features f a Furthermore, its global ID feature z is obtained through the branch module in S1.3. id ; Frame-by-frame extraction using facial expression motion estimation layers. Motion characteristics With z id The data is then merged and further analyzed using the head motion angle R of each frame corresponding to the source video. (i) Conditional delivery of motion adaptation adjustment layer M adp Obtain motion features of each frame corresponding to the source video ID and head motion alignment. Depend on Estimated optical flow motion representation Will With head movement angle R (i) Sequentially, volumetric deformation acts on the facial features f a Thus, the complete facial features after expression transfer are obtained. This feature enables facial expression transfer via the generative network G. Regarding the migration results Frames corresponding to the source video The final migration result is output by performing Laplace multi-level fusion based on the appropriate non-binary mask involved in S1.4.

[0157] Laplace multi-level fusion, also known as Laplacian pyramid multi-level fusion or multi-band fusion, is an advanced image processing technique. Based on the construction of the Laplacian pyramid, it decomposes an image into different frequency components and performs fusion operations on these components, ultimately reconstructing the fused image. The core of Laplace multi-level fusion lies in its ability to finely process high-frequency and low-frequency information in an image, thereby achieving high-quality image fusion results.

[0158] The construction of the Laplacian pyramid involves downsampling and upsampling operations on the image. First, the original image is downsampled using a Gaussian pyramid, resulting in a lower-resolution image after each downsampling step. Then, each layer of the Laplacian pyramid is obtained by calculating the difference between adjacent Gaussian pyramid layers. These difference images contain high-frequency information of the image, namely edges and details.

[0159] In Laplace multi-level fusion, a Laplacian pyramid is first constructed for each image to be fused. Then, at each level of the pyramid, a fusion operation is performed according to specific fusion rules. These fusion rules can be designed and adjusted according to specific application scenarios and requirements. For example, methods such as weighted averaging, maximum value selection, or minimum value selection can be used for fusion.

[0160] During the fusion process, the fusion of high-frequency components requires special care to avoid ghosting and truncation. Therefore, a smaller fusion window is typically used for high-frequency components to preserve image detail. For low-frequency components, which represent large areas of color in the image, a larger fusion window can be used to achieve a smoother fusion effect.

[0161] After fusing each layer, the fused image needs to be reconstructed through upsampling and addition operations. During this process, it is crucial to ensure that the fusion results of each layer are correctly superimposed to form the final high-quality fused image.

[0162] First, Laplace multi-level fusion can be used to fuse video frames, achieving smooth transitions between different video segments. By constructing a Laplace pyramid of video frames and performing fusion operations at each level, a more natural and fluid video transition effect can be obtained.

[0163] Secondly, Laplace multi-level blending can also be used for video special effects. By blending different images or video clips, various unique visual effects can be created, such as image splicing and background replacement. These effects have wide applications in film and television production, advertising production, and other fields.

[0164] Furthermore, Laplace multi-level fusion can also be used for video super-resolution reconstruction. By fusing multiple low-resolution video frames, high-resolution video images can be reconstructed, thereby improving the clarity and quality of the video. This is of great significance for fields such as video surveillance and medical imaging.

[0165] Practical Application Example 1:

[0166] In this exemplary embodiment, the recording of an online classroom lecture is used as an example for illustration. A recording video of an ordinary lecturer is used as the source video, and a video of a famous teacher or a lecture with expressive delivery is used as the target video as the input. The motion sequence features of the faces of the people in the target video with rich facial expressions are extracted. ID characteristics of the lecturer's image id Integration, and its effect on the facial features of the lecturer's image. a The video expression transfer model, after being pre-trained, is then further fused with the original lecturer's video at multiple levels to produce a lecture video output with richer expressions from the lecturer.

[0167] Practical Application Example 2:

[0168] In this exemplary embodiment, the production of game character cutscenes is used as an example for explanation. A source video of a game character displaying a blank or basic expression in a cutscene is selected, and a video of a professional actor delivering an emotionally rich performance is used as the target video. First, the motion sequence features of the actor's face in the target video are extracted. Then, it is compared with the game character's ID characteristic z. id The features are then fused. Next, the fused features are applied to the facial features representation of the game character. a The system uses a pre-trained facial expression transfer model to dynamically deform the character's facial expressions. Finally, the multi-layered rendering and expression fusion effects are applied to the game character's face in the source video, resulting in richer and more natural facial expressions in the game character's cutscenes.

[0169] It should be noted that the above examples 1 (model training), 2 (video expression transfer), practical application example 1, and practical application example 2 are only partial applications of the technology of this embodiment, and are not a complete expansion of the technology of this embodiment. They can be understood as references for the application of the technology of this embodiment in practical situations.

[0170] This embodiment has the following advantages:

[0171] By unifying the encoding of natural facial expressions, problems such as facial wriggling, texture discontinuities, and frame skipping during dynamic video frame generation can be largely avoided. The fusion of separate motion coding and post-processed ID information effectively decouples motion from character IDs, increases the adaptability of motion across different characters, and eliminates facial deformation issues caused by driving different characters. Employing volumetric feature representation in optical flow deformation significantly improves the naturalness of the transfer effect, while implicit motion representation enriches the expression details. Through motion adaptation adjustment layers and related training strategies, refined processing of facial rendering fusion is effectively achieved, eliminating potential edge imperfections and misalignment with the original video when the face is finally presented in the complete video area.

Claims

1. A method of zero-shot video-to-video expression transfer, the method comprising: The method comprises: acquiring a source video and an expression video, a face in the source video being a to-be-migrated object of a face in the expression video; preprocessing the source video and the expression video to obtain a target source video and a target expression video; performing video expression migration according to a target face global feature, a target motion feature, and a target expression migration model, to obtain a migrated video; the target face global feature is obtained by calculating the target source video through the target expression migration model; the target motion feature is obtained by calculating the target expression video through the target expression migration model; the video expression migration comprises face feature global analysis, face motion analysis, face volume deformation, and rendering processing; the target expression migration model comprises a face facial expression motion estimation module; the face facial expression motion estimation module comprises: a first feature encoding unit and a second feature encoding unit, wherein the first feature encoding unit is multiple, and the multiple first feature encoding units are in a stacked distribution; the first feature encoding unit comprises: a feature grouping subunit configured to divide an input image sequence into multiple first sequences; the image sequence comprises an image sequence obtained based on the target expression video or the target source video, and / or an image sequence output by a previous level first feature encoding unit; a feature processing subunit comprising multiple feature processing layers in a parallel distribution, and the multiple feature processing layers are configured to respectively perform feature encoding processing on the multiple first sequences to respectively output corresponding second sequences; a feature fusion subunit configured to perform feature fusion on the multiple second sequences to obtain a first image feature; the second feature encoding unit is configured to perform feature encoding processing on the first image feature output by each first feature encoding unit to obtain a second image feature; the face facial expression motion estimation module is further configured to obtain the target motion feature or the target face global feature according to the first image feature and the second image feature; the target expression migration model further comprises: a feature fusion module configured to perform feature fusion on the target face global feature and the target motion feature to obtain a target fusion feature; a coordinate module configured to import the target fusion feature into a regularized coordinate system and perform face volume deformation on the target fusion feature in the regularized coordinate system to obtain a target migration feature; a rendering module configured to perform rendering processing on the target migration feature to obtain a target migrated video; the coordinate module is further configured to: perform face volume deformation on the target fusion feature in the regularized coordinate system by using a head rotation angle of a corresponding target source video, the target migration feature having the same size as a face feature.

2. The method of zero-shot video-to-video expression transfer according to claim 1, wherein, The face facial expression motion estimation module is configured to perform face analysis on the target expression video and the target source video respectively to obtain the target motion feature and the target face global feature.

3. The method of zero-shot video-to-video expression transfer according to claim 2, wherein, The feature processing subunit further comprises a plurality of attention fusion layers, and each of the plurality of attention fusion layers corresponds to one of the plurality of feature processing layers; The attention fusion layer is configured to calculate attention weights according to the first sequence and the second sequence output by another feature processing layer, and perform feature fusion processing on the first sequence and the second sequence according to the attention weights; The feature processing layer is further configured to perform feature encoding processing on the fusion result of the first sequence and the second sequence to output the second sequence corresponding to the feature processing layer.

4. The method of zero-shot video-to-video expression transfer according to claim 1, wherein, The training process of the target expression transfer model comprises: obtaining training videos, wherein the training videos at least include a first training video and a second training video; preprocessing the first training video and the second training video respectively to obtain a first video and a second video; training a model according to the first video, the second video and an expression transfer model to obtain the target expression transfer model.

5. The method of zero-shot video-to-video expression transfer according to claim 4, wherein, The step of preprocessing the first training video and the second training video respectively comprises: performing initial feature extraction, image cropping and image organization screening on the first training video in sequence to obtain the first video; cropping images of the second training video to obtain the second video.

6. The method of zero-shot video-to-video expression transfer according to claim 5, wherein, The initial feature extraction comprises sequentially performing face position detection, face key point extraction and head rotation angle extraction on the first training video to obtain a face image and face data; the face data comprises a plurality of face key points and a head rotation angle; cropping images of the first training video comprises aligning a center point of a face in the face image with a center point of the face image based on the face key points, and performing non-transformation uniform cropping to obtain a cropped image; The image organization screening comprises screening according to the cropped image to obtain a first video, wherein the first video comprises a plurality of frames of the face in a natural state.

7. The method of zero-shot video-to-video expression transfer according to claim 5, wherein, The step of training a model according to the first video, the second video and an expression transfer model comprises: performing face feature analysis on the first video to obtain face global features; performing face motion analysis on the second video to obtain motion features; fusing the motion features and the face global features to obtain fused features; performing volumetric deformation on the fused features to obtain transfer features, and rendering the transfer features to obtain a transfer video; repeating the training until the transfer video meets the requirements of a preset loss function to obtain the target expression transfer model.

8. The method of zero-shot video-to-video expression transfer according to claim 7, wherein, The step of performing volumetric deformation on the fused features and rendering comprises: rendering the transfer features and setting a non-binary mask to limit the rendering process to obtain a transfer video until the transfer video meets the requirements of a loss function to obtain the target expression transfer model.

9. A system for zero-shot video-to-video expression transfer, the system comprising: The system is used to execute the method of any one of claims 1 to 8, and the system comprises: The data acquisition module (10) is configured to acquire a source video and an expression video, wherein a face in the source video is a to-be-migrated object of a face in the expression video. The preprocessing module (20) is configured to pre-process the source video and the expression video to obtain a target source video and a target expression video. The target expression migration module (30) is constructed based on a target expression migration model, and is configured to perform video expression migration according to a target face global feature, a target motion feature and the target expression migration model to obtain a migrated video; the target face global feature is obtained by calculating the target source video through the target expression migration model; the target motion feature is obtained by calculating the target expression video through the target expression migration model; and the video expression migration includes face feature global analysis, face motion analysis, face volume deformation and rendering processing.

Images