Method and system for migrating from zero sample video to video expression

Abstract

The invention provides a method and system for migrating a zero sample video to a video expression, and the method comprises the steps: obtaining a source video and an expression video, and enabling a human face in the source video to be a to-be-migrated object of the human face in the expression video; preprocessing the source video and the expression video to obtain a target source video and a target expression video; performing video expression migration according to the target face global feature, the target motion feature and a target expression migration model to obtain a migrated video; the target face global feature is obtained by calculating the target source video through the target expression migration model; the target motion feature is obtained by calculating the target expression video through the target expression migration model; the video expression migration comprises face feature extraction, face feature global analysis, face motion analysis, face volume deformation and rendering processing. According to the scheme, the problem that the expression migration effect of an existing video migration technology is poor is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of video expression migration, and in particular to a method and system for zero-sample video to video expression migration. Background Art

[0002] The study of facial expression transfer is an important topic in the field of computer vision and graphics, and is widely used in fields such as virtual reality, filmmaking, game development, and human-computer interaction. The core of facial expression transfer technology is to transfer the expression features of the source face to the target face, so that the target face can generate the same dynamic facial expressions as the source face. Usually, this technology relies on feature extraction of the source face, combined with deep learning or traditional graphics methods, to generate the corresponding expression changes of the target face. By using expression transfer technology, real-time expression synchronization of different faces can be achieved, thereby improving the realism and interactive experience of virtual characters.

[0003] The technology of transferring facial expressions between videos faces a series of technical challenges, mainly including the following aspects:

[0004] Accuracy of expression feature extraction: In order to achieve high-quality expression transfer, the expression features of the source face must be accurately extracted. However, due to the complexity of the face, different lighting conditions, and occlusion problems, the extracted expression features may be disturbed, resulting in unsatisfactory transfer results.

[0005] Handling cross-face differences: The facial structures of different individuals vary greatly, such as facial proportions, skin color, gender, age, etc., which poses additional challenges to expression transfer. How to accurately and naturally map expressions between different faces is an important technical problem.

[0006] Temporal consistency and coherence: For dynamic videos or real-time interactions, expression migration not only needs to process static expressions in each frame, but also needs to ensure temporal consistency during the migration process to avoid facial expressions jumping or unnatural transitions in consecutive frames. In particular, during the migration between videos, it is necessary to ensure that only the facial expressions are changed without affecting the head movement of the source video, while preventing a series of unnatural problems such as jitter caused by this.

[0007] Micro-expression capture: In addition to large facial expressions, the delicate capture and transmission of micro-expressions is also an important factor in improving the fidelity of migration. Current technologies still have difficulties in processing small, short-term changes in expression, especially in low-resolution videos or under suboptimal shooting conditions.

[0008] Preservation of details and texture: During the expression transfer process, maintaining the facial details and skin texture of the target face without distortion or interference from the source face is another key challenge. Especially under large changes in expression, facial details are prone to distortion or blurring. Summary of the invention

[0009] The present application provides a method and system for zero-sample video to video expression migration, so as to solve the problem that the existing video migration technology has poor expression migration effect.

[0010] In a first aspect, the present application provides a method for zero-sample video to video expression migration, the method comprising:

[0011] Obtaining a source video and an expression video, wherein the face in the source video is the object to be migrated of the face in the expression video;

[0012] Preprocessing the source video and the expression video to obtain a target source video and a target expression video;

[0013] Video expression migration is performed according to the target face global features, target motion features and target expression migration model to obtain a migration video; the target face global features are obtained by calculating the target source video via the target expression migration model; the target motion features are obtained by calculating the target expression video via the target expression migration model; the video expression migration includes global analysis of facial features, facial motion analysis, facial volume deformation and rendering processing.

[0014] Preferably, the target expression transfer model includes:

[0015] A face facial expression motion estimation module is configured to perform face motion analysis according to the target expression video and the target source video respectively to obtain the target face global features and the target motion features.

[0016] Preferably, the face facial expression motion estimation module comprises:

[0017] A first feature coding unit and a second feature coding unit, wherein the first feature coding unit is multiple and the multiple first feature coding units are arranged in a stacked manner;

[0018] The first feature encoding unit comprises:

[0019] A feature grouping subunit, wherein the feature grouping subunit is configured to divide an input image sequence into a plurality of first sequences; the image sequence includes an image sequence obtained based on the target expression video and / or an image sequence output by the first feature encoding unit at a previous level;

[0020] A feature processing subunit, the feature processing subunit comprising a plurality of feature processing layers that are distributed in parallel, the plurality of feature processing layers being configured to respectively perform feature encoding processing on the plurality of first sequences to respectively output corresponding second sequences;

[0021] a feature fusion subunit, configured to perform feature fusion according to a plurality of said second sequences to obtain a first image feature;

[0022] The second feature encoding unit is configured to perform feature encoding processing according to the first image feature output by each of the first feature encoding units to obtain the second image feature;

[0023] The face facial expression motion estimation module is also configured to obtain the target motion feature or the target face global feature according to the first image feature and the second image feature.

[0024] Preferably, the feature processing subunit further includes a plurality of attention fusion layers, and the plurality of attention fusion layers correspond one-to-one to the plurality of feature processing layers;

[0025] The attention fusion layer is configured to: calculate attention weights according to the first sequence and the second sequence output by another feature processing subunit, and perform feature fusion processing on the first sequence and the second sequence according to the attention weights;

[0026] The feature processing layer is further configured to perform feature encoding processing according to a fusion result of the first sequence and the second sequence to output the second sequence.

[0027] Preferably, the target expression transfer model also includes:

[0028] A feature fusion module, wherein the feature fusion module is configured to perform feature fusion according to the target face global feature and the target motion feature to obtain a target fusion feature;

[0029] A coordinate module, wherein the coordinate module is configured to import the target fusion feature into a regularized coordinate system, and perform face volume deformation on the target fusion feature in the regularized coordinate system to obtain a target migration feature;

[0030] A rendering module is configured to perform rendering processing according to the target migration characteristics to obtain a target migration video.

[0031] Preferably, the coordinate module is further configured as follows:

[0032] The fusion feature is volume-deformed using the head rotation angle corresponding to the first training video in the regularized coordinate system, and the size of the migration feature is the same as the facial feature.

[0033] Preferably, the training process of the target expression transfer model includes:

[0034] Acquire training videos, where the training videos at least include a first training video and a second training video;

[0035] Preprocessing the first training video and the second training video respectively to obtain a first video and a second video;

[0036] Model training is performed based on the first video, the second video and the expression transfer model to obtain the target expression transfer model.

[0037] Preferably, the step of preprocessing the first training video and the second training video respectively comprises:

[0038] Performing initial feature extraction, image cropping, and image organization screening on the first training video in sequence to obtain the first video;

[0039] Perform image cropping on the second training video to obtain the second video.

[0040] Preferably, the initial feature extraction includes sequentially performing face position detection, facial key point extraction, and head rotation angle extraction according to the first training video to obtain the face image and face data; the face data includes a number of facial key points and head rotation angles;

[0041] The image cropping includes aligning the center point of the face in the face image with the center point of the face image based on the facial key points, and performing uniform cropping without any deformation to obtain a cropped image;

[0042] The image organization screening includes screening according to the cropped image to obtain a first video, wherein the first video includes a plurality of frames of a human face in a natural state.

[0043] Preferably, the step of performing model training according to the first video, the second video and the expression transfer model comprises:

[0044] Performing facial feature analysis based on the first video to obtain global facial features;

[0045] Performing facial motion analysis based on the second video to obtain motion features;

[0046] Fusing the motion feature with the global face feature to obtain a fused feature;

[0047] Performing volume deformation according to the fusion feature to obtain a migration feature, and rendering the migration feature to obtain a migration video;

[0048] The training is repeated until the migration video meets the requirements of the preset loss function to obtain the target expression migration model.

[0049] Preferably, the performing volume deformation and rendering according to the fusion feature comprises:

[0050] The migration feature is rendered and a non-binary mask is set to limit the rendering process to obtain a migration video, until the migration video meets the requirements of the loss function, thereby obtaining the target expression migration model.

[0051] In a second aspect, the present application further provides a system for zero-sample video to video expression migration, the system being used to execute the above method, the system comprising:

[0052] A data acquisition module, wherein the data acquisition module is configured to acquire a source video and an expression video, wherein a human face in the source video is an object to be migrated of a human face in the expression video;

[0053] A preprocessing module, wherein the preprocessing module is configured to preprocess the source video and the expression video to obtain a target source video and a target expression video;

[0054] A target expression migration module, wherein the target expression migration module is constructed based on a target expression migration model, and the target expression migration module is configured to perform video expression migration according to the target face global features, target motion features and the target expression migration model to obtain a migration video; the target face global features are obtained by calculating the target source video via the target expression migration model; the target motion features are obtained by calculating the target expression video via the target expression migration model; the video expression migration includes global analysis of facial features, facial motion analysis, facial volume deformation and rendering processing.

[0055] As can be seen from the above content, the present application provides a method and system for zero-sample video to video expression migration, the method comprising obtaining a source video and an expression video, wherein the face in the source video is the object to be migrated of the face in the expression video; preprocessing the source video and the expression video to obtain a target source video and a target expression video; performing video expression migration according to the global features of the target face, the target motion features and the target expression migration model to obtain a migration video; the global features of the target face are obtained by calculating the target source video through the target expression migration model; the target motion features are obtained by calculating the target expression video through the target expression migration model; the video expression migration includes face feature extraction, global face feature analysis, face motion analysis, face volume deformation and rendering processing. The present application solves the problem of poor expression migration effect of existing video migration technology through the above scheme. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0057] Figure 1 A flowchart of a method for zero-sample video to video expression migration for this application;

[0058] Figure 2 A schematic diagram of a target expression transfer model in a method for zero-sample video to video expression transfer in this application;

[0059] Figure 3 A training flow chart of a target expression transfer model in a method for zero-sample video to video expression transfer in this application;

[0060] Figure 4 A flowchart of training video preprocessing in a method for zero-sample video to video expression transfer in this application;

[0061] Figure 5 A flowchart for specifically training an expression transfer model in a method for zero-sample video to video expression transfer in this application;

[0062] Figure 6 A relationship diagram between a target expression transfer model and processed data in a method for zero-sample video to video expression transfer in this application;

[0063] Figure 7 A schematic diagram of a system for zero-sample video to video expression migration in this application;

[0064] Figure 8 A schematic diagram of a facial expression motion estimation module in a zero-sample video to video expression transfer method of the present application;

[0065] Fig. 9 A schematic diagram of a first feature encoding unit in a zero-sample video to video expression migration method of the present application. DETAILED DESCRIPTION

[0066] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] Figure 1A flowchart of a method for zero-sample video to video expression migration is provided for this application.

[0068] See also Figure 1 It can be seen that this embodiment provides a method for zero-sample video to video expression migration, the method comprising:

[0069] S100, obtain the source video and the expression video. Specifically, in this embodiment, before performing video expression migration, it is necessary to first obtain the expression migration video and the expression providing video, wherein the source video is the expression migration video, and the expression video is the expression providing video. Therefore, it can be understood that the face in the source video is the object to be migrated of the face in the expression video.

[0070] It should be noted that, in this embodiment, video expression migration can be completed through expression migration video and expression providing video, and there is no need to adapt the human face in the process of video expression migration, so video expression migration with zero sample video can be achieved.

[0071] The method further comprises:

[0072] S200, preprocessing the source video and the expression video to obtain a target source video and a target expression video. Specifically, in the present embodiment, the source video and the expression video need to be preprocessed before performing the video expression migration. Since the obtained source video and / or the expression video may have problems such as non-standard face position, excessive facial expression amplitude, low video clarity, etc., they need to be preprocessed before the subsequent video expression migration process can be performed.

[0073] The method further comprises:

[0074] S300, performing video expression migration according to the target face global features, target motion features and target expression migration model to obtain a migration video. Specifically, in this embodiment, video expression migration is performed on the target face global features and target motion features through the pre-trained target expression migration model, wherein, during the expression migration process, the migrated face has been combined with the source video, and therefore there is no need to migrate the face after the expression migration.

[0075] In this embodiment, the facial features in the source video are combined as the facial framework reference, and the target video is combined with the motion prediction of the face to achieve the migration of facial expressions. The target facial global features are obtained by calculating the target source video through the target expression migration model; the target motion features are obtained by calculating the target expression video through the target expression migration model; the video expression migration includes facial feature extraction, global facial feature analysis, facial motion analysis, facial volume deformation and rendering processing.

[0076] It should be noted that the preprocessing is also completed in the target expression transfer model, so there is no need to perform prior processing on the acquired source video and the expression video, so as to improve the efficiency of expression transfer.

[0077] Figure 2 This is a schematic diagram of a target expression transfer model in a method for zero-sample video to video expression transfer in the present application.

[0078] See also Figure 2 It can be seen that, further, in some embodiments, the target expression transfer model includes:

[0079] A face facial expression motion estimation module is configured to perform face motion analysis according to the target expression video to obtain the target motion features and the target source video. Specifically, in this embodiment, the face facial expression motion estimation module performs face estimation on the target expression video or the target source video to obtain expression movements that are closer to a real face, and thereby cooperate with the features of the face to achieve smooth expression switching of the migrated face.

[0080] Figure 8 This is a schematic diagram of a facial expression motion estimation module in a zero-sample video to video expression migration method of the present application.

[0081] See also Figure 8 It can be seen that, further, in some embodiments, the face facial expression motion estimation module includes:

[0082] A first feature coding unit and a second feature coding unit, wherein the first feature coding unit is multiple and the multiple first feature coding units are arranged in a stacked manner;

[0083] The first feature encoding unit comprises:

[0084] A feature grouping subunit, wherein the feature grouping subunit is configured to divide an input image sequence into a plurality of first sequences; the image sequence includes an image sequence obtained based on the target expression video and / or an image sequence output by the first feature encoding unit at a previous level;

[0085] A feature processing subunit, the feature processing subunit comprising a plurality of feature processing layers that are distributed in parallel, the plurality of feature processing layers being configured to respectively perform feature encoding processing on the plurality of first sequences to respectively output corresponding second sequences;

[0086] a feature fusion subunit, configured to perform feature fusion according to a plurality of said second sequences to obtain a first image feature;

[0087] The second feature encoding unit is configured to perform feature encoding processing according to the first image feature output by each of the first feature encoding units to obtain the second image feature;

[0088] The face facial expression motion estimation module is also configured to obtain the target motion feature or the target face global feature according to the first image feature and the second image feature.

[0089] Specifically, in this embodiment, the face facial expression motion estimation module completes the face analysis of the target source video or the target expression video through the first feature encoding unit and the second feature encoding unit.

[0090] Among them, the local features in the video processing process are correspondingly optimized through the first feature encoding unit, and the global features in the image processing process are correspondingly optimized through the second feature encoding unit; the analysis and extraction of local and global features are realized through the first feature encoding unit and the second feature encoding unit, so that the effect of extracting the target motion features or the target face global features can be further improved.

[0091] Among them, the feature processing subunit with multiple levels of feature processing layers in the first feature encoding unit is used to perform feature encoding on different image sequences, and the encoded sequences are fused to realize the extraction of local features; the second feature encoding unit is used to further extract features from the sequence output by the first feature encoding unit to obtain global features.

[0092] It should be noted that, since the face facial expression motion estimation module has the function of extracting features from both the target source video and the target expression video, and the processing methods for different video sources are slightly different; for the target expression video that provides expressions, it is necessary to utilize the face facial expression motion estimation module to extract features that are more inclined to static features of the face based on the target expression video; for the target source video used to undertake expressions, it is necessary to utilize the face facial expression motion estimation module to extract features that are more inclined to dynamic features of the face based on the target source video.

[0093] Fig. 9 A schematic diagram of a first feature encoding unit in a zero-sample video to video expression migration method of the present application.

[0094] See also Fig. 9 It can be seen that, further, in some embodiments, the feature processing subunit further includes a plurality of attention fusion layers, and the plurality of attention fusion layers correspond one-to-one to the plurality of feature processing layers;

[0095] The attention fusion layer is configured to: calculate attention weights according to the first sequence and the second sequence output by another feature processing subunit, and perform feature fusion processing on the first sequence and the second sequence according to the attention weights;

[0096] The feature processing layer is further configured to perform feature encoding processing according to a fusion result of the first sequence and the second sequence to output the second sequence.

[0097] Specifically, in this embodiment, considering that the local features extracted by the first feature encoding unit may still contain some subtle features that cannot be accurately extracted, this embodiment sets the attention fusion layer in the first feature encoding unit, and cooperates with the feature processing layer through multiple attention fusion layers, so as to provide a more accurate technical basis for the extraction of local features.

[0098] Among them, by adding an attention weight mechanism to the sequence output by the feature processing subunit, the distribution of features in the sequence is made more consistent with the facial features of the actual face, so as to further improve the authenticity of the face after migration and avoid the sense of disobedience in the generated face. Furthermore, in some embodiments, the target expression migration model also includes:

[0099] The feature fusion module is configured to perform feature fusion according to the global features of the target face and the target motion features to obtain the target fusion features. Specifically, in this embodiment, since the face is relatively complex and the facial expressions are often switched during the facial expression migration, and the expression change cannot be smoothly switched by a single feature or motion prediction, in this embodiment, the feature fusion module is referenced to perform feature fusion on the global features of the target face and the target motion features, thereby obtaining the target fusion features having both the features of the face in the source video and the expression movements in the expression video. It can be understood that the key points of the two videos are combined through the feature fusion module to improve the video quality after the video expression migration.

[0100] The target expression transfer model also includes:

[0101] A coordinate module, wherein the coordinate module is configured to import the target fusion feature into a regularized coordinate system, and perform facial volume deformation on the target fusion feature in the regularized coordinate system to obtain a target migration feature. Specifically, in the present embodiment, since the parameters of the faces in different videos are different, it is necessary to perform a certain deformation on the fused target fusion feature, and placing the feature in a specific coordinate for deformation is conducive to the completion of the deformation. Therefore, in the present embodiment, the target fusion feature is imported into a regularized coordinate system through the coordinate module, and the facial volume deformation is performed on the target fusion feature in the regularized coordinate system to obtain the target migration feature that conforms to the face size after expression migration.

[0102] It should be noted that since the human face is not a single 2D image but a 3D structure, attention should be paid to the deformation of the face volume during the deformation process.

[0103] It should be noted that, in the regularized coordinate system, the volume deformation of the fusion feature is performed using the head rotation angle corresponding to the first training video, and the size of the migration feature is the same as the facial feature.

[0104] The target expression transfer model also includes:

[0105] A rendering module is configured to perform rendering processing according to the target migration feature to obtain a target migration video. Specifically, in the present embodiment, after obtaining the target migration feature, the single target migration feature cannot be directly used as a migration expression, and a corresponding video can only be generated after a certain rendering is performed according to the target migration feature. Therefore, the present embodiment introduces the rendering module, and the target migration feature is rendered by the rendering module to obtain the target migration video after the expression migration.

[0106] Figure 3 This is a training flowchart of a target expression transfer model in a method for zero-sample video to video expression transfer in this application.

[0107] See also Figure 3 It can be seen that, further, in some embodiments, the training process of the target expression transfer model includes:

[0108] S010, obtaining training videos, wherein the training videos include at least a first training video and a second training video. Specifically, in the present embodiment, similar to step S100, before training the expression transfer model, it is also necessary to obtain video samples for training, wherein the obtained video samples can be divided into two categories, one is a video for providing expressions, and the other is a video for waiting for expression transfer. In the present embodiment, the first training video is a video for waiting for expression transfer, and the second training video is a video for providing expressions.

[0109] The training process of the target expression transfer model also includes:

[0110] S020, preprocessing the first training video and the second training video respectively to obtain a first video and a second video. Specifically, in this embodiment, different preprocessing methods are used for different videos.

[0111] Figure 4 This is a flowchart of training video preprocessing in a method for zero-sample video to video expression migration in this application.

[0112] See also Figure 4 It can be seen that the steps of preprocessing the first training video and the second training video respectively include:

[0113] S021, performing initial feature extraction, image cropping, and image organization screening on the first training video in sequence to obtain the first video;

[0114] S022: Perform image cropping on the second training video to obtain the second video.

[0115] Specifically, in this embodiment, for the first training video, it is necessary to perform initial feature extraction, image cropping and image organization screening on the first training video in sequence to obtain the first video; for the second training video, it is only necessary to perform image cropping on the second training video to obtain the second video.

[0116] Among them, the initial feature extraction includes performing face position detection, facial key point extraction and head rotation angle extraction in sequence according to the first training video to obtain the face image and face data; the face data includes a number of facial key points and head rotation angles; the image cropping includes aligning the center point of the face in the face image with the center point of the face image based on the facial key points, and performing uniform cropping without deformation to obtain a cropped image; the image organization screening includes screening according to the cropped image to obtain a first video, and the first video includes a number of frames of the face in a natural state.

[0117] It should be noted that, when performing formal expression transfer, the preprocessing of the target video is the same as the preprocessing of the second training video, and the preprocessing of the source video is the same as the preprocessing of the first training video.

[0118] S030, performing model training according to the first video, the second video and the expression transfer model to obtain the target expression transfer model. Specifically, in this embodiment, model training is performed by importing the first video and the second video into the expression transfer model, wherein the expression transfer model is the target expression transfer model that has not been trained or has not completed training.

[0119] It should be noted that the training of the expression transfer model cannot be completed by using only the first training video and the second training video for training. It is necessary to use several videos that are the same as the first training video and the second training video for training several times to obtain the target expression transfer model.

[0120] Figure 5 A flowchart for specifically training an expression transfer model in a method for zero-sample video to video expression transfer in this application.

[0121] See also Figure 5 It can be seen that, further, in this embodiment, the step of performing model training according to the first video, the second video and the expression transfer model includes:

[0122] S031, performing facial feature analysis based on the first video to obtain global facial features;

[0123] S032, performing facial motion analysis according to the second video to obtain motion features;

[0124] S033, fusing the motion feature with the global face feature to obtain a fused feature;

[0125] S034, performing volume deformation according to the fusion feature to obtain a migration feature, and rendering the migration feature to obtain a migration video;

[0126] S035, repeat the training until the migration video meets the requirements of the preset loss function, and obtain the target expression migration model.

[0127] Specifically, in the present embodiment, the model training steps S031 to S035 are similar to step S300, which first extract the facial features of the expression video to be migrated; perform a global analysis on the facial features to obtain the global facial features; perform facial motion prediction on the expression video to obtain the motion features; combine the global facial features and the motion features, and perform relevant rendering on them to obtain the video after the expression migration; the difference is that for the model training process, it is necessary to undergo multiple expression migrations to obtain the final target expression migration model.

[0128] Among them, in the process of volume deformation and rendering according to the fusion features, it is necessary to render the migration features and set a non-binary mask to limit the rendering process to obtain the migration video, until the migration video meets the requirements of the loss function, thereby obtaining the target expression migration model.

[0129] It should be noted that obtaining the source video and the expression video, preprocessing the source video and the expression video respectively, obtaining the first training video and the second training video, and preprocessing the first training video and the second training video respectively all have one thing in common, that is, the acquisition or preprocessing of the two videos are carried out simultaneously.

[0130] Figure 7 A schematic diagram of a system for zero-sample video to video expression migration according to the present application.

[0131] See also Figure 7 It can be seen that this embodiment also provides a system for zero-sample video to video expression migration, the system is used to execute the above method, and the system includes:

[0132] The data acquisition module 10 is configured to acquire a source video and an expression video, wherein the human face in the source video is an object to be migrated of the human face in the expression video. Specifically, in the present embodiment, the data acquisition module 10 is used to acquire the source video and the expression video, but it should be noted that the source video and the expression video are both video sources in actual expression migration, and during the model training process, the data acquisition module 10 can also acquire the video source for training.

[0133] The system further comprises:

[0134] A preprocessing module 20, wherein the preprocessing module 20 is configured to preprocess the source video and the expression video to obtain a target source video and a target expression video. Specifically, in the present embodiment, the preprocessing module 20 is used to preprocess the source video and the expression video. It should be noted that the types of preprocessing performed on the source video and the expression video are different, and preprocessing may not be performed depending on the actual video conditions of the source video and the expression video. Among them, similar to the above-mentioned data acquisition module 10, the preprocessing module 20 is not only used in the actual application of expression migration, but can also be used in the process of model training.

[0135] The system further comprises:

[0136] A target expression migration module 30 is constructed based on a target expression migration model, and is configured to perform video expression migration according to target face global features, target motion features and a target expression migration model to obtain a migration video; the target face global features are obtained by calculating the target source video via the target expression migration model; the target motion features are obtained by calculating the target expression video via the target expression migration model; the video expression migration includes global facial feature analysis, facial motion analysis, facial volume deformation and rendering processing.

[0137] Specifically, in this embodiment, the target expression migration module 30 extracts features from the target source video and the target expression video respectively, and fuses the two features together to achieve expression migration; wherein the target expression migration module 30 simultaneously extracts local features and global features of the image, and fuses the two, thereby improving the authenticity of the face after expression migration.

[0138] Figure 6 This is a relationship diagram between a target expression transfer model and processed data in a method for zero-sample video to video expression transfer in this application.

[0139] See also Figure 6 It can be seen that this embodiment also provides two specific implementation methods: model training and video expression transfer.

[0140] The principle of the solution of this embodiment is as follows:

[0141] First, estimate and extract the natural appearance features of the face in the current frame of the source video (f a ), estimate the face pose of the source video, estimate the expression motion of the target video, and calculate the expression motion features of the target video (f e ), combined with the source video face pose (f p ) and the target video expression motion feature (f e ) for fa Implement deformation to obtain deformed facial features The facial rendering fusion module is used to generate the migrated face and render it into the face area in the source video frame.

[0142] Example 1 (model training):

[0143] S1.1 Training data preprocessing mainly includes four parts: data collection, feature extraction, data cropping, and data organization and screening. Data collection uses any high-definition shooting equipment to ensure that the face can be clearly photographed. The shooting content should include clips of people speaking, multi-angle, and rich expressions on the face. Feature extraction mainly includes the detection of face position, extraction of facial key points, and estimation of head rotation angle; data cropping is mainly the process of aligning the center point of the face with the center position of the image frame by frame based on key points for invisible uniform cropping. The main purpose of data organization and screening is to filter out natural expression facial frames and balance the distribution of each head movement and expression category. This process mainly combines the distribution of facial key points, the distance ratio coefficient of the key points related to the eye area and the mouth area, and the classification of facial expressions to classify and organize the facial attributes of each video frame, and then balance the results, and filter out the natural facial expression frames separately. It should be noted that in the scenario of the present invention, we define the natural facial expression frame as: detecting the head movement yaw angle (Yaw) in the interval (-10°, 10°), the roll angle (Roll) in the interval (-8°, 8°), the pitch angle (Pitch) in the interval (-10°, 10°), the eye opening and closing ratio in the interval (0.3, 0.5), and the mouth opening and closing ratio (5e -3 , 2e -2 ), the expression is classified as neutral with a confidence level of more than 80%, and the confidence levels of the other categories do not exceed 45%.

[0144] S1.2 Pre-training facial features estimation and extraction module for natural facial expressions, using the residual convolutional network block (ResBlock) as the basic module to build an autoencoder (E A , D A ), taking any frame of any video clip as input, and outputting the target (groundtruth, gt) as any natural facial expression frame corresponding to the video. In this process, we define the following objective function:

[0145]

[0146] in They respectively represent the input frame features obtained by the encoder part of the video frame after passing through the encoder and the features of its corresponding natural facial expression frame. The final decoder output reconstruction for the input frame, I neuis the input natural facial expression frame. It should be noted that the features here are 3D volumetric features, and the decoder in the autoencoder is a decoder network structure with projection characteristics. At the same time, the decoder is a coarse decoder and is only used for auxiliary training.

[0147] S1.3 End-to-end training of facial expression motion estimation, motion deformation construction, and facial rendering fusion. The facial expression motion estimation module is a network structure improved based on the existing CNN. Through this network, we obtain the latent space feature representation z of facial expression motion. m ; For the 3D facial features mentioned in S1.2, a facial global feature z is extracted through a branch module based on the existing CNN improvement at this stage id ; z m With z id Fusion obtains the motion semantic representation corresponding to the current character's face ID In the motion deformation construction stage, a 1x1 convolution structure is used to represent the global motion It is expanded into a 2x2x2 volume representation with spatial local structure, and the representation is expanded to the same size as f through multi-level 3D residual convolution blocks. a The volume representation of the same scale (natural appearance features) is used to obtain a body motion estimate relative to the regularized coordinate system through the additionally defined prediction layer and offset calculation. Specifically, the body motion estimate can be understood as an implicit 3D optical flow representation corresponding to the regularized coordinate system. Move your facial expressions The head rotation angle R of the input frame is detected, and the volume deformation is applied to the standardized facial representation f of the input frame in turn. a , thus obtaining the expression movement z m Complete facial features with head movement angle R Adopt D a Double the number of network layers to build the rendering fusion module G, so that The complete image of the current input frame after expression migration is obtained through the output of network G. In this stage S1.2, E A Will be frozen and not participate in training. The input and reconstruction target are two different frames in the same video segment k. Specifically, the above a Should come from Motion-related features and head motion R should be extracted from At this stage, we use the following objective function to constrain the entire network:

[0148]

[0149] in express Migrate to The reconstruction loss, To combat losses, To reconstruct the id feature constraints of the back, It is the reuse of the objective function in S1.2, which is used to control the distribution consistency of the standard facial appearance representation. λ1 and λ2 are adjustable hyperparameters. Through this process, the texture-consistent migration motion effect can be achieved while ensuring sufficient facial expression details.

[0150] S1.4 Refinement of facial rendering fusion, through this process to ensure that the actual current frame can be seamlessly integrated into the current complete video frame image after the facial region expression migration is completed. Under the premise of S1.3 training completion, an additional motion adaptation adjustment layer M is introduced in the facial expression motion estimation layer adp At this stage, only M adp Train and fine-tune G. Use hetero-id video clips as input Provides as source the id face representation f a With head moving R, Provide z as an expression target m , after the migration process as S1.3, but making Under condition R, a motion adaptation adjustment layer M is applied. adp The rest of the process is the same as described in S1.3. After completing one expression transfer, the output is We define an appropriate non-binary mask for Constrain the area so that its outer edges are reconstructed Internal reconstruction does not pass M adp Migration output at We define the objective function of this stage as follows:

[0151]

[0152] in represents the reconstruction of the area outside the mask, represents the reconstruction of the internal area of ​​the mask, represents a self-reconstruction consistent with that in S1.3, is the continuity constraint of the motion representation, λ c This stage method can solve various inadaptability problems in video migration in terms of pasting back and migration motion continuity.

[0153] Among them, non-binary masks, as an image processing technology, are often used to control the display and hiding of specific areas in layers or video images. The grayscale value of non-binary masks ranges from 0 (completely transparent) to 255 (completely opaque), where gray areas represent varying degrees of translucency. This type of mask is widely used in film and television editing and image processing. For example, non-binary masks can be used to achieve smooth transitions between video clips, or to create various complex overlay and gradient effects. In the video generation process, non-binary masks can help creators control the display area of ​​the video screen more finely, thereby achieving a more natural and smooth visual effect.

[0154] Example 2 (Video Expression Migration):

[0155] S2.1 Input video preprocessing: Given a source video First, the source video is preprocessed by cropping the face area and center alignment, and the head rotation parameters of each frame are extracted. Select a target expression video The face area of ​​the target expression video is cropped, and the cropping process adopts the fixed frame coordinate cropping method.

[0156] S2.2 Model Reasoning: Input encoder E a Encode to get the appearance feature f a , and further obtain its global id feature z through the branch module in S1.3 id ; Use facial expression motion estimation layer to extract frame by frame Movement characteristics With z id Fusion, and further correspond to the source video frame head motion angle R (i) The condition is fed into the motion adaptation adjustment layer M adp , obtain the motion features of each frame corresponding to the source video id and head movement alignment Depend on Estimating Optical Flow Motion Estimation Representation Will Head Angle R (i) After volume deformation, the appearance characteristics f a , thus obtaining the complete features of the face after expression transfer This feature is used to generate the facial image through the generative network G. Migration results Corresponding frame to source video Based on the appropriate non-binary mask mentioned in S1.4, Laplace multi-level fusion is performed to achieve the output of the final migration result

[0157] Among them, Laplace multi-level fusion, also known as Laplace pyramid multi-level fusion or multi-band fusion, is an advanced image processing technology. It is based on the construction of Laplace pyramid, decomposes the image into different frequency components, performs fusion operations on these components, and finally reconstructs the fused image. The core of Laplace multi-level fusion is that it can finely process the high-frequency and low-frequency information in the image, thereby achieving high-quality image fusion effects.

[0158] The construction process of the Laplacian pyramid involves downsampling and upsampling operations of the image. First, the original image is downsampled through the Gaussian pyramid, and each downsampling results in a lower resolution image. Then, each layer of the Laplacian pyramid is obtained by calculating the difference between two adjacent layers of the Gaussian pyramid images. These difference images contain the high-frequency information of the image, that is, the edges and details in the image.

[0159] In Laplace multi-level fusion, it is first necessary to construct Laplace pyramids for the images to be fused. Then, on each level of the pyramid, fusion operations are performed according to specific fusion rules. These fusion rules can be designed and adjusted according to specific application scenarios and requirements. For example, weighted averaging, maximum value selection, or minimum value selection can be used for fusion.

[0160] During the fusion process, the fusion of high-frequency components needs to be particularly careful to avoid ghosting and truncation. Therefore, a smaller fusion window is usually used for high-frequency components to preserve the details in the image. For low-frequency components, since they represent large color blocks in the image, a larger fusion window can be used to achieve a smooth fusion effect.

[0161] After completing the fusion of each layer, it is necessary to reconstruct the fused image through upsampling and addition operations. In this process, it is necessary to ensure that the fusion results of each layer can be correctly superimposed together to form the final high-quality fused image.

[0162] First, Laplace multi-level fusion can be used to fuse video frames to achieve smooth transitions between different video clips. By constructing a Laplace pyramid of video frames and performing fusion operations on each layer, a more natural and smooth video transition effect can be obtained.

[0163] Secondly, Laplace multi-level fusion can also be used to produce video special effects. By fusing different images or video clips, you can create a variety of unique visual effects, such as image splicing, background replacement, etc. These special effects have a wide range of application value in film and television production, advertising production and other fields.

[0164] In addition, Laplace multi-level fusion can also be used for video super-resolution reconstruction. By fusing multiple low-resolution video frames, a high-resolution video image can be reconstructed, thereby improving the clarity and quality of the video. This is of great significance in the fields of video surveillance, medical imaging, etc.

[0165] Practical application example 1:

[0166] In this exemplary embodiment, an online class recording is used as an example to illustrate. A recorded video of an ordinary lecturer is used as the source video, and a video of a famous teacher or a lecture with a good mood is used as the target video as input. By extracting the motion sequence features of the face of the character in the target video with rich expression, the target video is used as the input. The id feature z of the lecturer image id Fusion, and acting on the lecturer's facial features a The pre-trained video expression transfer model is further fused with the original lecturer video at multiple levels to obtain the lecture video output with richer expressions of the lecturer.

[0167] Practical application example 2:

[0168] In this exemplary embodiment, the production of game character cutscenes is used as an example. A source video of a game character with no expression or basic expression in the cutscene is selected, and a professional actor's emotional performance video is used as the target video. First, the motion sequence features of the actor's face in the target video are extracted. Then with the game character's id feature z id Then the fused features are applied to the facial features of the game character. a , the character's facial expression is dynamically deformed through the pre-trained expression transfer model. Finally, the multi-level rendering and expression fusion effects are applied to the game character's face in the source video, making the game character show richer and more natural expression output in the cutscenes.

[0169] It should be noted that the above-mentioned Example 1 (model training), Example 2 (video expression transfer), actual application example 1 and actual application example 2 are only partial applications of the technology of this embodiment, and are not the full development of the technology of this embodiment. They can be understood as references for the application of the technology of this embodiment in actual situations.

[0170] This embodiment has the following advantages:

[0171] The unified encoding of natural facial expressions can greatly avoid problems such as facial crawling, texture discontinuity, and frame skipping in the process of generating dynamic video frames. The fusion of separate motion coding and post-ID information can effectively decouple motion from character ID, increase the adaptability of motion on different characters, and eliminate the problem of facial deformation caused by driving between different characters. The use of volumetric feature representation to participate in optical flow deformation greatly improves the naturalness of the migration effect. At the same time, the implicit motion representation enriches the expression details. Through the motion adaptation adjustment layer and its related training strategies, the refined processing of facial rendering fusion is effectively achieved, eliminating a series of problems that may appear when the face is finally presented in the complete video area, such as edge defects and non-fitting with the original video.

Claims

1. A method for zero-sample video to video expression migration, characterized in that: The method comprises: Obtaining a source video and an expression video, wherein the face in the source video is the object to be migrated of the face in the expression video; Preprocessing the source video and the expression video to obtain a target source video and a target expression video; Video expression migration is performed according to the target face global features, target motion features and target expression migration model to obtain a migration video; the target face global features are obtained by calculating the target source video via the target expression migration model; the target motion features are obtained by calculating the target expression video via the target expression migration model; the video expression migration includes global analysis of facial features, facial motion analysis, facial volume deformation and rendering processing.

2. The method for zero-sample video to video expression migration according to claim 1, characterized in that: The target expression transfer model includes: A face facial expression motion estimation module is configured to perform face analysis according to the target expression video and the target source video respectively to obtain the target face global features and the target motion features.

3. The method for zero-sample video to video expression migration according to claim 2, characterized in that: The face expression motion estimation module comprises: A first feature coding unit and a second feature coding unit, wherein the first feature coding unit is multiple and the multiple first feature coding units are arranged in a stacked manner; The first feature encoding unit comprises: A feature grouping subunit, wherein the feature grouping subunit is configured to divide an input image sequence into a plurality of first sequences; the image sequence includes an image sequence obtained based on the target expression video and / or an image sequence output by the first feature encoding unit at a previous level; A feature processing subunit, the feature processing subunit comprising a plurality of feature processing layers that are distributed in parallel, the plurality of feature processing layers being configured to respectively perform feature encoding processing on the plurality of first sequences to respectively output corresponding second sequences; a feature fusion subunit, configured to perform feature fusion according to a plurality of said second sequences to obtain a first image feature; The second feature encoding unit is configured to perform feature encoding processing according to the first image feature output by each of the first feature encoding units to obtain the second image feature; The face facial expression motion estimation module is also configured to obtain the target motion feature or the target face global feature according to the first image feature and the second image feature.

4. The method for zero-sample video to video expression migration according to claim 3, characterized in that: The feature processing subunit further includes a plurality of attention fusion layers, and the plurality of attention fusion layers correspond one-to-one to the plurality of feature processing layers; The attention fusion layer is configured to: calculate attention weights according to the first sequence and the second sequence output by another feature processing subunit, and perform feature fusion processing on the first sequence and the second sequence according to the attention weights; The feature processing layer is further configured to perform feature encoding processing according to a fusion result of the first sequence and the second sequence to output the second sequence.

5. The method for zero-sample video to video expression migration according to claim 3, characterized in that: The target expression transfer model also includes: A feature fusion module, wherein the feature fusion module is configured to perform feature fusion according to the target face global feature and the target motion feature to obtain a target fusion feature; A coordinate module, wherein the coordinate module is configured to import the target fusion feature into a regularized coordinate system, and perform face volume deformation on the target fusion feature in the regularized coordinate system to obtain a target migration feature; A rendering module is configured to perform rendering processing according to the target migration characteristics to obtain a target migration video.

6. The method for zero-sample video to video expression migration according to claim 5, characterized in that: The coordinate module is further configured as: The fusion feature is volume-deformed using the head rotation angle corresponding to the first training video in the regularized coordinate system, and the size of the migration feature is the same as the facial feature.

7. The method for zero-sample video to video expression migration according to claim 1, characterized in that: The training process of the target expression transfer model includes: Acquire training videos, where the training videos at least include a first training video and a second training video; Preprocessing the first training video and the second training video respectively to obtain a first video and a second video; Model training is performed based on the first video, the second video and the expression transfer model to obtain the target expression transfer model.

8. The method for zero-sample video to video expression migration according to claim 7, characterized in that: The step of preprocessing the first training video and the second training video respectively comprises: Performing initial feature extraction, image cropping, and image organization screening on the first training video in sequence to obtain the first video; Perform image cropping on the second training video to obtain the second video.

9. The method for zero-sample video to video expression migration according to claim 8, characterized in that: The initial feature extraction includes sequentially performing face position detection, facial key point extraction, and head rotation angle extraction according to the first training video to obtain the face image and face data; the face data includes a number of facial key points and head rotation angles; The image cropping includes aligning the center point of the face in the face image with the center point of the face image based on the facial key points, and performing uniform cropping without any deformation to obtain a cropped image; The image organization screening includes screening according to the cropped image to obtain a first video, wherein the first video includes a plurality of frames of a human face in a natural state.

10. The method for zero-sample video to video expression migration according to claim 8, characterized in that: The step of performing model training according to the first video, the second video and the expression transfer model comprises: Performing facial feature analysis based on the first video to obtain global facial features; Performing facial motion analysis based on the second video to obtain motion features; Fusing the motion feature with the global face feature to obtain a fused feature; Performing volume deformation according to the fusion feature to obtain a migration feature, and rendering the migration feature to obtain a migration video; The training is repeated until the migration video meets the requirements of the preset loss function to obtain the target expression migration model.

11. The method for zero-sample video to video expression migration according to claim 10, characterized in that: The performing volume deformation and rendering according to the fusion feature comprises: The migration feature is rendered and a non-binary mask is set to limit the rendering process to obtain a migration video, until the migration video meets the requirements of the loss function, thereby obtaining the target expression migration model.

12. A system for zero-sample video to video expression migration, characterized in that: The system is used to perform the method according to any one of claims 1 to 11, and the system comprises: A data acquisition module (10), wherein the data acquisition module (10) is configured to acquire a source video and an expression video, wherein a human face in the source video is an object to be migrated of a human face in the expression video; A preprocessing module (20), wherein the preprocessing module (20) is configured to preprocess the source video and the expression video to obtain a target source video and a target expression video; A target expression migration module (30), the target expression migration module (30) is constructed based on a target expression migration model, the target expression migration module (30) is configured to perform video expression migration according to the target face global features, target motion features and the target expression migration model to obtain a migration video; the target face global features are obtained by calculating the target source video through the target expression migration model; the target motion features are obtained by calculating the target expression video through the target expression migration model; the video expression migration includes global facial feature analysis, facial motion analysis, facial volume deformation and rendering processing.

Images