A method and apparatus for facial motion and expression transfer with detailed optimization compensation

Abstract

This invention discloses a method and apparatus for facial motion and expression transfer with detail optimization and compensation. The invention uses a 3D facial reconstruction model to model and encode facial motion and expression, achieving facial information acquisition and compression. A sliding window is used to ensure the consistency of the characters in the generated video. Multiple deep information fusion techniques are used to obtain the facial pixel flow field, initially distorting the original image to address the problems of poor generalization ability and missing motion and expression in keypoint-based expression transfer models for faces with significant differences. Multi-level facial detail encoding and optimization compensation models are used to specifically repair missing parts of the image, solving problems such as detail loss and edge distortion caused by information deficiency.

Description

Technical Field

[0001] This invention relates to the field of computer vision information technology, and in particular to a method and apparatus for facial motion expression transfer with detail optimization compensation. Background Technology

[0002] With the development of deep learning and computer vision, generating realistic speaking head videos has attracted widespread attention. Speaking head video generation aims to use motion information extracted from target-driven video or audio to animate the dynamic poses and expressions of faces in still images while preserving the identity of the person in the source image. This technique can "bring" historical figures to life, enabling them to speak and perform some head movements based on the provided motion information.

[0003] In recent years, spoken head video generation can be broadly categorized into two approaches based on the driving medium: image-driven and audio-driven. Image-based methods encompass a range of techniques that utilize still images or sequences of frames as primary input, employing sophisticated machine learning algorithms to animate visual content, simulate speech, changing facial expressions, and capture subtle movements. In contrast, audio-based techniques use audio input to synthesize corresponding facial movements, with a particular focus on lip movements and emotional expressions consistent with the audio input.

[0004] Image-driven techniques can be broadly categorized into two types: those using 2D information correlation (ICR) and those using 3D ICR. 2D ICR primarily uses keypoints to model motion flow fields. Some methods use pre-trained facial keypoint detection models to acquire keypoints. These methods typically obtain a large number of keypoints (e.g., 68), but due to errors in the pre-trained model, the modeled flow field also inherits these errors, leading to error accumulation. Furthermore, since the keypoints are located primarily on the face, the motion modeling effect for other parts such as clothing and accessories is poor. Other methods, on the other hand, design complex keypoint detection networks to detect the locations of keypoints in the video stream that undergo displacement changes. These locations (usually few, only 10-20) cover the entire image, improving the motion modeling effect. Keypoint-based methods share a common problem: due to the differences in facial structure across tasks, using keypoints from two different individuals to model the flow field results in the generated image's facial structure tending to favor the driving image rather than preserving the source image's facial structure. Therefore, it's necessary to use the relative displacement of keypoints between the two driving images to model the flow field. While this method preserves the source image's facial structure, variations in the magnitude and amplitude of the relative displacement lead to incomplete motion in the generated image (e.g., half-closed eyes). Furthermore, in the presence of facial occlusion, keypoint detection exhibits significant errors, causing motion modeling to fail. 3D information-related techniques improve transfer performance by introducing 3D information from the image. These methods utilize well-designed generative networks to obtain depth and 3D reconstruction parameters from the face image, enhancing transfer performance by adding extra information. However, these methods are limited by the 3D information generation network; introducing extra information also introduces noise, requiring careful network structure design. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by proposing a method and apparatus for facial motion and expression transfer with detailed optimization and compensation.

[0006] The objective of this invention is achieved through the following technical solution: a method for facial motion expression transfer with detailed optimization compensation, the method comprising:

[0007] S1. Obtain the video dataset and perform frame-by-frame preprocessing;

[0008] S2. Randomly select two frames from a video as the source image and driving image of the video, match the number of facial parameters in the source image and driving image, and use the facial parameters in the two images to construct the final driving parameters.

[0009] S3. Construct a pixel flow field extraction network model with the final driving parameters and source image as inputs;

[0010] S4. Construct a detail optimization compensation neural network model with the source image as input, including downsampling the source image to obtain the original features and distorting the feature map through the pixel flow field, upsampling the distorted feature map, and performing detail optimization compensation between the original features obtained by each downsampling layer and the distorted features obtained by each upsampling layer during the upsampling process.

[0011] S5. Train the constructed network model: Calculate the difference between the result image and the driving image, and use gradient descent for backpropagation training;

[0012] S6. Take the first frame of the video to be processed as the source image and the remaining frames as the driving images. Use the trained network model to generate the video and obtain the transferred video after detail optimization and compensation.

[0013] Furthermore, the frame-segmentation preprocessing includes: extracting facial parameter information for each video frame, including identity parameters, expression parameters, texture parameters, rotation angle parameters, lighting parameters, translation parameters, and cropping parameters; and modifying the color order and resolution after segmenting the video into frames.

[0014] Furthermore, the step of matching the number of facial parameters in the source image and the driving image specifically involves: using the parameters of the source image frame as the source parameters, using the parameters of the number of sliding window frames before and after the driving image frame as the driving parameters, and copying the number of source parameters to be the same as the number of driving parameters.

[0015] Furthermore, the construction of the final driving parameters specifically involves replacing the identity parameters in the driving parameters with the identity parameters in the source parameters, retaining the expression, rotation angle, and translation parameters in the driving parameters, and combining them into the final driving parameters.

[0016] Furthermore, the pixel flow field extraction network model specifically includes:

[0017] Using several one-dimensional convolutions and LeakyReLU activation functions, and then through adaptive average pooling, the final driving parameters with a shape adjusted to [bs,256,1] are obtained;

[0018] The pixel flow field is obtained using the source image and the final driving parameters after shape adjustment. This includes three layers of downsampling followed by three layers of upsampling using the downsampling results to obtain the pixel flow field feature map. This map is then processed sequentially through two 2D convolutional layers, 2D batch normalization, LeakyReLU, and another 2D convolutional layer to obtain the pixel flow field and adjust it to a standard pixel flow field.

[0019] Each of the three downsampling layers includes: adaptive instance normalization, LeakyReLU, two-dimensional convolutional layer, adaptive instance normalization, and two-dimensional convolutional layer. The adaptive instance normalization integrates the encoding parameters into the original image, and the convolutional kernel in the first two-dimensional convolutional layer is larger than that in the second two-dimensional convolutional layer.

[0020] Each of the three upsampling layers includes: processing the downsampling result or the upsampling result of the previous layer through adaptive instance normalization, LeakyReLU, and a two-dimensional deconvolution layer to obtain the retained result;

[0021] The downsampling result or the upsampling result from the previous layer is processed through adaptive instance normalization, LeakyReLU, a 2D convolutional layer, adaptive instance normalization, LeakyReLU, and 2D deconvolution to obtain the fused result; the retained result and the fused result are added together, and then channel-level concatenation is performed with the source image.

[0022] Furthermore, the downsampling in the detailed optimization compensation neural network model includes four layers; the first downsampling layer includes: two-dimensional convolution, instance normalization, and ReLU activation function; the remaining downsampling layers include two-dimensional convolution, instance normalization, ReLU activation function, and average pooling.

[0023] Furthermore, the step of distorting the feature map through pixel flow field specifically involves: for the fourth-layer downsampled feature map, firstly, using PyTorch's interpolation method F.interpolate to modify the width and height of the pixel flow field to the width and height of the fourth-layer downsampled feature map through bilinear interpolation, and then using PyTorch's sampling method F.grid_sample to sample the feature map through the pixel flow field to obtain the distorted fourth-layer feature map.

[0024] Furthermore, the upsampling of the distorted feature map is specifically as follows:

[0025] The distorted fourth-layer feature map is upsampled using F.interpolate, 2D convolution, instance normalization, and ReLU activation function to obtain the third-layer upsampled feature map; detail optimization compensation is performed in each subsequent upsampled layer to obtain the first-layer detail optimization compensation feature map, which is then processed by 2D convolution and sigmoid activation function to obtain the final result;

[0026] The detailed optimization compensation in each upsampling layer includes: using the upsampled feature map obtained from the previous layer as the distorted feature, and using the downsampled feature map corresponding to that layer as the original feature.

[0027] The distortion features are segmented at the channel level, into image features that need to be retained and image features that need to be optimized;

[0028] The original features are distorted using pixel flow fields, and targeted detail feature maps are obtained using 2D convolution, instance normalization, and ReLU activation functions.

[0029] The image features to be optimized are processed by two-dimensional convolution, instance normalization and ReLU activation function to obtain the missing detail feature map, and then channel-level connection is made with the targeted detail feature map.

[0030] The concatenated feature map is processed through 2D convolution, instance normalization, and sigmoid activation function to obtain a mask;

[0031] The missing detail feature map is multiplied by the mask, and the targeted detail feature map is multiplied by 1-mask to obtain the detail compensation feature map;

[0032] The detail compensation feature map is connected to the image features to be preserved at the channel level, and then processed by two-dimensional convolution, instance normalization and ReLU activation function, and then processed by two residual connection layers to obtain the detail optimization compensation feature map;

[0033] The detailed optimization compensation feature map is upsampled by F.interpolate, 2D convolution, instance normalization, and ReLU activation function to obtain the upsampled feature map for input to the next layer.

[0034] Furthermore, the training of the constructed network model specifically involves:

[0035] The extracted training data is sequentially passed through a neural network model for obtaining pixel flow fields and a detail optimization and compensation neural network model. The difference between the resulting image and the driving image is calculated. Gradient descent is used for backpropagation to train the entire model. The differences between the resulting image and the driving image include:

[0036] The result image and the driving image are downsampled three times using two-dimensional anti-aliasing interpolation. The number of channels remains unchanged in each downsampling, and the image width and height are reduced to half of the original. These images are then combined with the original image to form the feature map of the result image and the feature map of the driving image, respectively.

[0037] For each pair of feature maps with the same width and height, calculate the inter-image differences, which include the following three parts:

[0038] Reconstruction loss: Calculate the L1 loss between two feature maps, which is the average of the absolute values ​​of the image differences;

[0039] Style loss: Calculate the L1 loss between the style matrices of the two feature maps, as follows:

[0040] The image is reshaped into a two-dimensional matrix, that is, the shape is reshaped from [c,w,h] to [c,w*h], where c is the number of channels, w is the width, and h is the height;

[0041] Multiplying this matrix by its transpose yields a matrix of shape [c, c], denoted as the style matrix.

[0042] The L1 loss between the style matrix of the result image feature map and the style matrix of the driving image feature map is calculated, which is the average of the absolute values ​​of the differences between corresponding elements of the matrices.

[0043] Perceptual Loss: Calculate the L1 loss between the perceptual features of two feature maps. The calculation method is as follows:

[0044] The perceptual features of the feature map were extracted using the VGG19 model, and the results of layers 2, 7, 12, 21, and 30 were taken as the perceptual features.

[0045] The L1 loss between the feature map of the result image and the perceptual features of the corresponding layer of the driving image feature map is calculated as the average of the absolute values ​​of the differences between corresponding element features.

[0046] According to another aspect of the specification, the present invention also provides a facial motion expression transfer device with detail optimization compensation, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the facial motion expression transfer method with detail optimization compensation.

[0047] The beneficial effects of this invention are:

[0048] 1. This invention utilizes facial parameters from a 3D reconstruction model to model the flow field, solving the problem of mismatch and incomplete motion modeling when modeling key points with relative displacements to facial structures that differ greatly from the actual facial structure. This model models motion flow based solely on facial pose and the source image, thereby eliminating noise generated by the facial structure information in the driving image.

[0049] 2. This invention solves the problems of keypoint modeling requiring similar initial poses, high requirements for driving video, and insufficient generalization. Because this model does not use relative displacement, it has no requirements for the initial pose of the driving video, can produce reasonable generation effects for any driving video, and has a wider range of applicability.

[0050] 3. This invention improves the generation results when the face in the driving video is occluded (e.g., by arms or microphones). Thanks to the robustness of the facial reconstruction model, even when the driving image is partially occluded, the model can still correctly extract facial motion parameters, thereby correctly modeling the flow field and generating the corresponding pose and expression.

[0051] 4. This invention utilizes the original image for detail optimization and compensation, solving the problem of information loss caused by distorted images alone. The generated result retains more of the original image's detailed features, thus improving fidelity. Attached Figure Description

[0052] Figure 1 This is a flowchart illustrating the process of obtaining combined parameters for this invention.

[0053] Figure 2 To obtain a pixel flow field flowchart for this invention;

[0054] Figure 3 The network flowchart for this invention has been optimized.

[0055] Figure 4 This is a flowchart of the detailed optimization module of the present invention;

[0056] Figure 5 This is a schematic diagram of the device of the present invention. Detailed Implementation

[0057] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0058] A facial motion expression transfer method with detailed optimization compensation includes:

[0059] S1. Training Dataset Preparation and Data Preprocessing: Prepare the video training set, extract the parameters required for training, and modify it to the required training format. This includes the following sub-steps:

[0060] (1) Obtain the VoxCeleb video dataset and divide it into training set and test set in a ratio of 20:1.

[0061] (2) Use the Deep3DFaceRecon_pytorch model to extract facial parameter information for each video frame, including identity parameters, expression parameters, texture parameters, rotation angle parameters, lighting parameters, translation parameters, and cropping parameters. Store the parameters in the parameter folder "data / 3dmms" and name it "person number-video number-segment number.npy";

[0062] (3) Process the video into frames, change the color order to RGB, use bilinear interpolation to change the resolution to 256 pixels wide and 256 pixels high, save it into the frame folder "data / pngs", and name it "character number-video number-segment number / frame number.png";

[0063] S2, such as Figure 1 As shown, two frames are randomly selected from a video as the source image and the driving image. The facial parameters in the source image and the driving image are matched in number, and the final driving parameters are constructed using the facial parameters in the two images. This includes the following sub-steps:

[0064] (1) Randomly select video numbers and randomly select two frames from the selected video as the source image and the driving image;

[0065] (2) Obtain the parameters of the source image frame from the face parameter file .npy of the corresponding video as the source parameters, and use the parameters of the sliding window (radius of 13) frames before and after the driving image frame as the driving parameters.

[0066] (3) Copy the number of source parameters to the same number of driving parameters, replace the identity parameters in the driving parameters with the identity parameters in the source parameters, retain the expression, rotation angle and translation parameters in the driving parameters, combine them into the final driving parameters, and discard the rest of the source parameters and driving parameters.

[0067] S3, such as Figure 2 The diagram shows the construction of a pixel flow field extraction network model that takes the final driving parameters and the source image as input. The steps of data processing in this model are as follows:

[0068] S3.1. Use a one-dimensional convolution (input features 230, output features 256, kernel 7, stride 1, padding 0) to adjust the shape of the combined parameters to [bs, 256, 21], where bs is the batch size;

[0069] S3.2. Use LeakyReLU (slope of 0.1) and one-dimensional convolution (input features of 256, output features of 256, kernel of 3, stride of 1, padding of 0, spacing of 3) to adjust the shape of the combined parameters to [bs, 256, 15], and add it to the middle 15 bits of the parameters before adjustment (i.e., [bs, 256, 3: -3]).

[0070] S3.3. Use LeakyRelu (slope of 0.1) and one-dimensional convolution (input features of 256, output features of 256, kernel of 3, stride of 1, padding of 0, spacing of 3) to adjust the shape of the combined parameters to [bs,256,9], and add it to the middle 15 bits of the parameters before adjustment (i.e., [bs,256,3:-3]).

[0071] S3.4. Use LeakyRelu (slope of 0.1) and one-dimensional convolution (input features of 256, output features of 256, kernel of 3, stride of 1, padding of 0, spacing of 3) to adjust the shape of the combined parameters to [bs, 256, 3], and add it to the middle 15 bits of the parameters before adjustment (i.e., [bs, 256, 3: -3]).

[0072] S3.5. Use adaptive average pooling to adjust the shape of the combined parameters to [bs, 256, 1] to obtain the encoding parameters;

[0073] S3.6 Obtaining the pixel flow field from the source image and encoding parameters includes the following sub-steps:

[0074] S3.6.1. Adaptive instance normalization is used to fuse the encoding parameters into the original image. After processing with LeakyReLU (slope of 0.1) and 2D convolution (3 input feature maps, 64 output feature maps, 4 kernels, 2 stride, and 1 padding), the encoding parameters are fused into the original image again using adaptive instance normalization. After processing with LeakyReLU (slope of 0.1) and 2D convolution (64 input feature maps, 64 output feature maps, 3 kernels, 2 stride, and 1 padding), the first layer downsampling result is obtained.

[0075] S3.6.2. Adaptive instance normalization is used to fuse the encoding parameters into the original image. After processing with LeakyReLU (slope of 0.1) and 2D convolution (64 input feature maps, 128 output feature maps, 4 kernels, 2 strides, and 1 padding), the encoding parameters are fused into the original image again using adaptive instance normalization. After processing with LeakyReLU (slope of 0.1) and 2D convolution (128 input feature maps, 128 output feature maps, 3 kernels, 2 strides, and 1 padding), the second layer downsampling result is obtained.

[0076] S3.6.3. Adaptive instance normalization is used to fuse the encoding parameters into the original image. After processing with LeakyReLU (slope of 0.1) and 2D convolution (128 input feature maps, 256 output feature maps, 4 kernels, 2 strides, and 1 padding), the encoding parameters are fused into the original image again using adaptive instance normalization. After processing with LeakyReLU (slope of 0.1) and 2D convolution (256 input feature maps, 256 output feature maps, 3 kernels, 2 strides, and 1 padding), the third layer downsampling result is obtained.

[0077] S3.6.4. For the third-layer downsampling result, adaptive instance normalization is used to fuse the encoding parameters, followed by LeakyReLU (slope of 0.1) and 2D deconvolution (input feature map number of 256, output feature map number of 128, convolution kernel number of 3, stride of 2, padding of 1, output padding of 1) to obtain the retained result; For the third-layer downsampling result, adaptive instance normalization is used to fuse the encoding parameters, followed by LeakyReLU (slope of 0.1) and 2D deconvolution (input feature map number of 256, output feature map number of 128, convolution kernel number of 3, stride of 2, padding of 1, output padding of 1) to obtain the retained result; The following parameters were used to obtain the fusion result: 256 input feature maps, 128 output feature maps, 3 kernels, 2 strides, and 1 padding; adaptive instance normalization fusion coding parameters; LeakyReLU (slope of 0.1); and 2D deconvolution (128 input feature maps, 128 output feature maps, 3 kernels, 2 strides, 1 padding, and 1 output padding). The retained result and the fusion result were added together, and then connected to the second-layer downsampling result at the channel level to obtain the second-layer upsampling result.

[0078] S3.6.5. For the second-layer upsampling result, adaptive instance normalization fusion encoding parameters are used, followed by LeakyReLU (slope 0.1) and 2D deconvolution (256 input feature maps, 64 output feature maps, 3 kernels, 2 stride, 1 padding, 1 output padding) to obtain the retained result; For the second-layer upsampling result, adaptive instance normalization fusion encoding parameters, LeakyReLU (slope 0.1), 2D convolution (256 input feature maps, 64 output feature maps, 3 kernels, 2 stride, 1 padding), adaptive instance normalization fusion encoding parameters, LeakyReLU (slope 0.1), 2D deconvolution (64 input feature maps, 64 output feature maps, 3 kernels, 2 stride, 1 padding, 1 output padding) are used to obtain the fused result; The retained result and the fused result are added together, and then connected to the first-layer downsampling result at the channel level to obtain the first-layer upsampling result;

[0079] S3.6.6. The first-layer upsampling result is processed using adaptive instance normalization fusion encoding parameters, followed by LeakyReLU (slope 0.1) and 2D deconvolution (128 input feature maps, 64 output feature maps, 3 kernels, 2 stride, 1 padding, 1 output padding) to obtain the retained result. The first-layer upsampling result is then processed using adaptive instance normalization fusion encoding parameters, LeakyReLU (slope 0.1), 2D convolution (128 input feature maps, 64 output feature maps, 3 kernels, 2 stride, 1 padding), adaptive instance normalization fusion encoding parameters, LeakyReLU (slope 0.1), and 2D deconvolution (64 input feature maps, 64 output feature maps, 3 kernels, 2 stride, 1 padding, 1 output padding) to obtain the fused result. The retained result and the fused result are added together and then connected to the source image at the channel level to obtain the pixel flow field feature map.

[0080] S3.6.7 The pixel flow field feature map is processed by two-dimensional convolution (67 input feature maps, 128 output feature maps, 4 kernels, 2 strides, and 1 padding), two-dimensional convolution (128 input feature maps, 256 output feature maps, 4 kernels, 2 strides, and 1 padding), two-dimensional batch normalization, LeakyReLU (slope of 0.1), and two-dimensional convolution (256 input feature maps, 2 output feature maps, 7 kernels, 1 stride, and 3 padding) to obtain the pixel flow field;

[0081] S3.6.8 Divide the first dimension of the pixel flow field by the width (64), divide the second dimension by the height (64), and add it to the standard coordinates (width is 64, height is 64, coordinate value is [-1,1]) that are the same as the shape of the pixel flow field to obtain the standard pixel flow field;

[0082] S4, such as Figure 3 , Figure 4 As shown in the diagram, a detail optimization and compensation neural network model is established with the source image as input. The steps of this model in processing the data are as follows:

[0083] S4.1. Image downsampling to obtain downsampled feature maps includes the following sub-steps:

[0084] (1) The source image is processed by two-dimensional convolution (3 input feature maps, 64 output feature maps, 3 convolution kernels, 1 stride, and 1 padding), instance normalization, and ReLU activation function to obtain the first layer downsampled feature map;

[0085] (2) The first-layer downsampled feature map is downsampled using two-dimensional convolution (64 input feature maps, 128 output feature maps, 3 convolution kernels, 1 stride, and 1 padding), instance normalization, ReLU activation function, and average pooling (2 convolution kernels) to obtain the second-layer downsampled feature map.

[0086] (3) The second-layer downsampled feature map is downsampled using two-dimensional convolution (128 input feature maps, 256 output feature maps, 3 convolution kernels, 1 stride, and 1 padding), instance normalization, ReLU activation function, and average pooling (2 convolution kernels) to obtain the third-layer downsampled feature map.

[0087] (4) The third-layer downsampled feature map is downsampled using two-dimensional convolution (256 input feature maps, 512 output feature maps, 3 convolution kernels, 1 stride, and 1 padding), instance normalization, ReLU activation function, and average pooling (2 convolution kernels) to obtain the fourth-layer downsampled feature map.

[0088] S4.2 For the fourth-layer downsampled feature map, first use PyTorch's interpolation method F.interpolate to modify the pixel flow field width and height to the width and height of the fourth-layer downsampled feature map through bilinear interpolation, and then use PyTorch's sampling method F.grid_sample to sample on the feature map through the pixel flow field to obtain the distorted fourth-layer feature map (this process is called "distorting the feature map using the pixel flow field").

[0089] S4.3 Upsampling and detailed optimization compensation, including the following sub-steps:

[0090] (1) The distorted fourth layer feature map is upsampled using F.interpolate (scaling factor of 2), two-dimensional convolution (512 input feature maps, 256 output feature maps, 3 convolution kernels, 1 stride, 1 padding), instance normalization, and ReLU activation function to obtain the third layer upsampled feature map.

[0091] (2) Perform detail optimization compensation between the third-layer downsampled feature map (denoted as the original feature) and the third-layer upsampled feature map (denoted as the distorted feature), including the following sub-steps:

[0092] ① Perform channel-level segmentation on the distorted features, dividing them into image feature IDs that need to be retained and image feature details that need to be optimized;

[0093] ② Use pixel flow field distortion of the original features, and use two-dimensional convolution (256 input channels, 256 output channels, 3 kernels, 1 stride, 1 padding), instance normalization and ReLU activation function to obtain targeted detail feature maps;

[0094] ③ The image features to be optimized are processed through two-dimensional convolution (128 input channels, 256 output channels, 3 kernels, 1 stride, and 1 padding), instance normalization, and ReLU activation function to obtain the missing detail feature map, which is then connected to the targeted detail feature map at the channel level.

[0095] ④ The connected feature map is processed through two-dimensional convolution (512 input channels, 256 output channels, 3 kernels, 1 stride, and 1 padding), instance normalization, and sigmoid activation function to obtain a mask;

[0096] ⑤ Multiply the missing detail feature map by the mask, and multiply the targeted detail feature map by (1-mask) to obtain the detail compensation feature map;

[0097] ⑥ The detail compensation feature map is connected to the image feature ID to be retained at the channel level. After passing through a two-dimensional convolution (384 input channels, 256 output channels, 3 kernels, 1 stride, and 1 padding), instance normalization, and ReLU activation function, two residual connection layers are used (instantly performing instance normalization, ReLU activation function, two-dimensional convolution (channel number and width and height remain unchanged), instance normalization, ReLU activation function, and two-dimensional convolution (channel number and width and height remain unchanged), and the result is added to the input) to obtain the third layer of detail optimization compensation feature map;

[0098] ⑦ The third-layer detail optimization compensation feature map is upsampled by F.interpolate (scaling factor of 2), two-dimensional convolution (256 input feature maps, 128 output feature maps, 3 convolution kernels, 1 stride, 1 padding), instance normalization, and ReLU activation function to obtain the second-layer upsampled feature map.

[0099] (3) Perform detail optimization compensation between the second-layer downsampled feature map (denoted as the original feature) and the second-layer upsampled feature map (denoted as the distorted feature), including the following sub-steps:

[0100] ① Perform channel-level segmentation on the distorted features, dividing them into image feature IDs that need to be retained and image feature details that need to be optimized;

[0101] ② Use pixel flow field distortion of the original features, and use two-dimensional convolution (128 input channels, 128 output channels, 3 convolution kernels, 1 stride, 1 padding), instance normalization and ReLU activation function to obtain targeted detail feature maps;

[0102] ③ The image features to be optimized are processed through two-dimensional convolution (64 input channels, 128 output channels, 3 kernels, 1 stride, and 1 padding), instance normalization, and ReLU activation function to obtain the missing detail feature map, which is then connected to the targeted detail feature map at the channel level.

[0103] ④ The connected feature map is processed through two-dimensional convolution (256 input channels, 128 output channels, 3 kernels, 1 stride, and 1 padding), instance normalization, and sigmoid activation function to obtain a mask;

[0104] ⑤ Multiply the missing detail feature map by the mask, and multiply the targeted detail feature map by (1-mask) to obtain the detail compensation feature map;

[0105] ⑥ The detail compensation feature map is connected to the image feature ID to be retained at the channel level. After passing through a two-dimensional convolution (192 input channels, 128 output channels, 3 kernels, 1 stride, and 1 padding), instance normalization, and ReLU activation function, two residual connection layers are used (instantly performing instance normalization, ReLU activation function, two-dimensional convolution (channel number and width and height remain unchanged), instance normalization, ReLU activation function, and two-dimensional convolution (channel number and width and height remain unchanged), and the result is added to the input) to obtain the second layer of detail optimization compensation feature map;

[0106] ⑦ The second-layer detail optimization compensation feature map is upsampled by F.interpolate (scaling factor of 2), two-dimensional convolution (128 input feature maps, 64 output feature maps, 3 convolution kernels, 1 stride, 1 padding), instance normalization, and ReLU activation function to obtain the first-layer upsampled feature map;

[0107] (4) Perform detail optimization compensation between the first-layer downsampled feature map (denoted as the original feature) and the first-layer upsampled feature map (denoted as the distorted feature), including the following sub-steps:

[0108] ① Perform channel-level segmentation on the distorted features, dividing them into image feature IDs that need to be retained and image feature details that need to be optimized;

[0109] ② Use pixel flow field distortion of the original features, and use two-dimensional convolution (64 input channels, 64 output channels, 3 convolution kernels, 1 stride, 1 padding), instance normalization and ReLU activation function to obtain targeted detail feature maps;

[0110] ③ The image features to be optimized are processed through two-dimensional convolution (32 input channels, 64 output channels, 3 kernels, 1 stride, and 1 padding), instance normalization, and ReLU activation function to obtain the missing detail feature map, which is then connected to the targeted detail feature map at the channel level.

[0111] ④ The connected feature map is processed through two-dimensional convolution (128 input channels, 64 output channels, 3 kernels, 1 stride, and 1 padding), instance normalization, and sigmoid activation function to obtain a mask;

[0112] ⑤ Multiply the missing detail feature map by (1-mask), and multiply the targeted detail feature map by the mask to obtain the detail compensation feature map;

[0113] ⑥ The detail compensation feature map is connected to the image feature ID to be retained at the channel level. After passing through a two-dimensional convolution (96 input channels, 64 output channels, 3 kernels, 1 stride, and 1 padding), instance normalization, and ReLU activation function, two residual connection layers are used (instantly performing instance normalization, ReLU activation function, two-dimensional convolution (channel number and width and height remain unchanged), instance normalization, ReLU activation function, and two-dimensional convolution (channel number and width and height remain unchanged), and the result is added to the input) to obtain the first layer of detail optimization compensation feature map;

[0114] S4.4 The first layer of detail optimization and compensation feature map is processed by two-dimensional convolution (64 input channels, 3 output channels, 3 convolution kernels, 1 stride, and 1 padding) and sigmoid activation function to obtain the final result.

[0115] S5. Training the entire neural network model: The extracted training data is sequentially passed through a neural network model for obtaining pixel flow fields and a detail optimization and compensation neural network model. The difference between the resulting image and the driving image is calculated, and gradient descent is used for backpropagation to train the entire model. The batch size is set to 12, and the model is trained for 200 rounds on the entire training set. The image difference calculation steps are as follows;

[0116] S5.1. Perform three downsampling operations on the result image and the driving image using two-dimensional anti-aliasing interpolation. The number of channels remains unchanged in each downsampling operation, and the image width and height are reduced to half of their original values. These downsampling operations are then combined with the original image to form the feature map of the result image and the feature map of the driving image, respectively.

[0117] S5.2 For each pair of feature maps with the same width and height, calculate the differences between the images. The differences between the images include the following three parts:

[0118] Reconstruction loss: Calculate the L1 loss between two feature maps, which is the average of the absolute values ​​of the image differences. The weight is 1.

[0119] Style loss: Calculate the L1 loss between the style matrices of the two feature maps, as follows:

[0120] ① Reshape the image into a two-dimensional matrix, that is, reshape the shape from [c,w,h] to [c,w*h], where c is the number of channels, w is the width, and h is the height.

[0121] ② Perform matrix multiplication between this matrix and its transpose to obtain a matrix of shape [c,c], denoted as the style matrix.

[0122] ③ Calculate the L1 loss between the style matrix of the resulting image feature map and the style matrix of the driving image feature map, which is the average of the absolute values ​​of the differences between corresponding elements of the matrices. The weight is 250.

[0123] Perceptual Loss: Calculate the L1 loss between the perceptual features of two feature maps. The calculation method is as follows:

[0124] ① Use the VGG19 model to extract the perceptual features of the feature map, and take the results of layers 2, 7, 12, 21 and 30 as the perceptual features.

[0125] ② Calculate the L1 loss between the feature map of the resulting image and the perceptual features of the corresponding layer of the driving image feature map, which is the average of the absolute values ​​of the differences between corresponding element features. The weight is 1.

[0126] S6. Validate on the test set. Use the test set videos as data, with the first frame as the source image and all video frames as driving images, to perform a video generation task and validate the model's performance.

[0127] Corresponding to the aforementioned embodiment of a facial motion expression transfer method with detail optimization compensation, the present invention also provides an embodiment of a facial motion expression transfer device with detail optimization compensation.

[0128] See Figure 5 The present invention provides a facial motion expression transfer device with detail optimization compensation, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement a facial motion expression transfer method with detail optimization compensation as described in the above embodiment.

[0129] An embodiment of the facial motion and expression transfer device with detailed optimization compensation provided by this invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 5 The diagram shown is a hardware structure diagram of any device with data processing capabilities, which is used in the facial motion and expression transfer device for detailed optimization compensation provided by the present invention. (Except for...) Figure 5 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0130] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0131] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0132] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements a facial motion expression transfer method with detailed optimization compensation as described in the above embodiments.

[0133] The computer-readable storage medium can be an internal storage unit of any data processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0134] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the aforementioned detailed optimization compensation method for facial motion expression transfer.

[0135] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0136] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. This application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for facial motion and expression transfer with detailed optimization compensation, characterized in that, The method includes: S1. Obtain the video dataset and perform frame-by-frame preprocessing; S2. Randomly select two frames from a video as the source image and driving image of the video, match the number of facial parameters in the source image and driving image, and use the facial parameters in the two images to construct the final driving parameters. S3. Construct a pixel flow field extraction network model with the final driving parameters and source image as inputs; S4. Construct a detail optimization compensation neural network model with the source image as input, including downsampling the source image to obtain the original features and distorting the feature map through the pixel flow field, upsampling the distorted feature map, and performing detail optimization compensation between the original features obtained by each downsampling layer and the distorted features obtained by each upsampling layer during the upsampling process. S5. Train the constructed network model: Calculate the difference between the output image and the driving image of the neural network model for detail optimization compensation, and use gradient descent for backpropagation training; S6. Take the first frame of the video to be processed as the source image and the remaining frames as the driving images. Use the trained network model to generate the video and obtain the transferred video after detail optimization and compensation.

2. The facial motion and expression transfer method with detailed optimization compensation according to claim 1, characterized in that, The frame-segmentation preprocessing includes: extracting facial parameter information for each video frame, including identity parameters, expression parameters, texture parameters, rotation angle parameters, lighting parameters, translation parameters, and cropping parameters; and modifying the color order and resolution using bilinear interpolation after segmenting the video.

3. The facial motion and expression transfer method with detailed optimization compensation according to claim 1, characterized in that, The specific steps of matching the number of facial parameters in the source image and the driving image are as follows: the parameters of the source image frame are used as source parameters, the parameters of the number of sliding window frames before and after the driving image frame are used as driving parameters, and the number of source parameters is copied to be the same as the number of driving parameters.

4. The facial motion expression transfer method with detailed optimization compensation according to claim 3, characterized in that, The specific steps for constructing the final driving parameters are as follows: replace the identity parameters in the driving parameters with the identity parameters in the source parameters, retain the expression, rotation angle, and translation parameters in the driving parameters, and combine them into the final driving parameters.

5. The facial motion and expression transfer method with detailed optimization compensation according to claim 1, characterized in that, The pixel flow field extraction network model specifically includes: The final driving parameters use several one-dimensional convolutions and LeakyReLU activation functions, and then adaptive average pooling is used to obtain the encoding parameters with a shape adjusted to [bs, 256, 1]. The pixel flow field is obtained using the source image and the shape-adjusted encoding parameters. This includes three layers of downsampling followed by three layers of upsampling using the downsampling results to obtain the pixel flow field feature map. This feature map is then processed sequentially through two 2D convolutional layers, 2D batch normalization, LeakyReLU, and another 2D convolutional layer to obtain the pixel flow field and adjust it to a standard pixel flow field. Each of the three downsampling layers includes: adaptive instance normalization, LeakyReLU, two-dimensional convolutional layer, adaptive instance normalization, and two-dimensional convolutional layer. The adaptive instance normalization integrates the encoding parameters into the original image, and the convolutional kernel in the first two-dimensional convolutional layer is larger than that in the second two-dimensional convolutional layer. The three-layer upsampling includes: The third-layer downsampling result is processed by adaptive instance normalization fusion encoding parameters, followed by LeakyReLU and 2D deconvolution to obtain the retained result. The third-layer downsampling result is then processed by adaptive instance normalization fusion encoding parameters, LeakyReLU, 2D convolution, adaptive instance normalization fusion encoding parameters, LeakyReLU, and 2D deconvolution to obtain the fused result. The retained result and the fused result are added together and then connected to the second-layer downsampling result at the channel level to obtain the second-layer upsampling result. The second-layer upsampling result is processed using adaptive instance normalization fusion encoding parameters, followed by LeakyReLU and 2D deconvolution to obtain the retained result. The second-layer upsampling result is then processed by adaptive instance normalization fusion encoding parameters, LeakyReLU, 2D convolution, adaptive instance normalization fusion encoding parameters, LeakyReLU, and 2D deconvolution to obtain the fused result. The retained result and the fused result are added together, and then channel-level concatenation is performed with the first-layer downsampling result to obtain the first-layer upsampling result. The first-layer upsampling result is processed by adaptive instance normalization fusion encoding parameters, followed by LeakyReLU and 2D deconvolution to obtain the retained result. The first-layer upsampling result is then processed by adaptive instance normalization fusion encoding parameters, LeakyReLU, 2D convolution, adaptive instance normalization fusion encoding parameters, LeakyReLU, and 2D deconvolution to obtain the fused result. The retained result and the fused result are added together and then channel-level concatenated with the source image to obtain the pixel flow field feature map.

6. The facial motion expression transfer method with detailed optimization compensation according to claim 1, characterized in that, The detailed optimization compensation neural network model includes four downsampling layers; the first downsampling layer includes: two-dimensional convolution, instance normalization, and ReLU activation function; the remaining downsampling layers include two-dimensional convolution, instance normalization, ReLU activation function, and average pooling.

7. The facial motion and expression transfer method with detailed optimization compensation according to claim 6, characterized in that, The process of distorting the feature map by pixel flow field is as follows: For the fourth-layer downsampled feature map, first use PyTorch's interpolation method F.interpolate to modify the width and height of the pixel flow field to the width and height of the fourth-layer downsampled feature map through bilinear interpolation, and then use PyTorch's sampling method F.grid_sample to sample the feature map through the pixel flow field to obtain the distorted fourth-layer feature map.

8. The facial motion expression transfer method with detailed optimization compensation according to claim 7, characterized in that, Upsampling the distorted feature map specifically involves: The distorted fourth-layer feature map is upsampled using F.interpolate, 2D convolution, instance normalization, and ReLU activation function to obtain the third-layer upsampled feature map; detail optimization compensation is performed in each subsequent upsampled layer to obtain the first-layer detail optimization compensation feature map, which is then processed by 2D convolution and sigmoid activation function to obtain the final result; The detailed optimization compensation in each upsampling layer includes: using the upsampled feature map obtained from the previous layer as the distorted feature, and using the downsampled feature map corresponding to that layer as the original feature. The distortion features are segmented at the channel level, into image features that need to be retained and image features that need to be optimized; The original features are distorted using pixel flow fields, and targeted detail feature maps are obtained using 2D convolution, instance normalization, and ReLU activation functions. The image features to be optimized are processed by two-dimensional convolution, instance normalization and ReLU activation function to obtain the missing detail feature map, and then channel-level connection is made with the targeted detail feature map. The concatenated feature map is processed through 2D convolution, instance normalization, and sigmoid activation function to obtain a mask; The missing detail feature map is multiplied by the mask, and the targeted detail feature map is multiplied by 1-mask to obtain the detail compensation feature map; The detail compensation feature map is connected to the image features to be preserved at the channel level, and then processed by two-dimensional convolution, instance normalization and ReLU activation function, and then processed by two residual connection layers to obtain the detail optimization compensation feature map; The detailed optimization compensation feature map is upsampled by F.interpolate, 2D convolution, instance normalization, and ReLU activation function to obtain the upsampled feature map for input to the next layer.

9. The facial motion expression transfer method with detailed optimization compensation according to claim 1, characterized in that, The specific steps for training the constructed network model are as follows: The extracted training data is sequentially passed through a neural network model for obtaining pixel flow fields and a detail optimization and compensation neural network model. The difference between the resulting image and the driving image is calculated. Gradient descent is used for backpropagation to train the entire model. The differences between the resulting image and the driving image include: The result image and the driving image are downsampled three times using two-dimensional anti-aliasing interpolation. The number of channels remains unchanged in each downsampling, and the image width and height are reduced to half of the original. These images are then combined with the original image to form the feature map of the result image and the feature map of the driving image, respectively. For each pair of feature maps with the same width and height, calculate the inter-image differences, which include the following three parts: Reconstruction loss: Calculate the L1 loss between two feature maps, which is the average of the absolute values ​​of the image differences; Style loss: Calculate the L1 loss between the style matrices of the two feature maps, as follows: The image is reshaped into a two-dimensional matrix, that is, the shape is reshaped from [c, w, h] to [c, w*h], where c is the number of channels, w is the width, and h is the height; Multiplying this matrix by its transpose yields a matrix of shape [c, c], denoted as the style matrix. The L1 loss between the style matrix of the result image feature map and the style matrix of the driving image feature map is calculated, which is the average of the absolute values ​​of the differences between corresponding elements of the matrices. Perceptual Loss: Calculate the L1 loss between the perceptual features of two feature maps. The calculation method is as follows: The perceptual features of the feature map were extracted using the VGG19 model, and the results of layers 2, 7, 12, 21, and 30 were taken as the perceptual features. The L1 loss between the feature map of the result image and the perceptual features of the corresponding layer of the driving image feature map is calculated as the average of the absolute values ​​of the differences between corresponding element features.

10. A facial motion expression transfer device with detailed optimization compensation, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements a facial motion expression transfer method with detailed optimization compensation as described in any one of claims 1-9.

Images