Method and device for generating facial animation based on prior knowledge identity preservation

Abstract

The present disclosure provides a method and device for generating facial animation based on prior knowledge identity preservation, which belongs to the technical field of computer graphics. The method comprises: extracting features from a source image to obtain source image features, extracting features from a driving image to obtain driving image features; calculating background optical flow of a target image based on the background of the source image and the driving image; obtaining motion optical flow of the target image based on the source image features, the driving image features and the background optical flow; and obtaining a target image in response to inputting the source image and the motion optical flow into an identity enhancement generator network. The method and device for generating facial animation based on prior knowledge identity preservation provided by the present disclosure can improve the visual experience of the audience.

Description

Technical Field

[0001] The present disclosure belongs to the technical field of computer graphics, and more specifically, to a method and device for generating facial animation based on prior knowledge identity preservation. Background Art

[0002] As one of the core research areas in computer graphics, computer vision and digital image processing, facial animation generation technology aims to generate high-fidelity, natural and smooth facial animations through video, audio or text information. With the vigorous development of virtual reality (VR) and augmented reality (AR) technologies, facial animation has shown great potential in film production, game development, remote video conferencing and human-computer interaction. However, traditional manual production methods are not only time-consuming and labor-intensive, but also costly, and it is difficult to meet modern requirements for efficiency and precision.

[0003] Currently, the video-driven facial animation generation method has become the mainstream technology because it can capture rich posture expression information and facial details. However, this method still faces many challenges in identity preservation. Most existing technologies fail to make full use of prior knowledge to effectively preserve the identity information of the source image, resulting in identity loss or posture distortion during the animation process, which seriously affects the realism of the animation and the audience's visual experience. Summary of the invention

[0004] The purpose of the present disclosure is to provide a method and device for generating facial animation based on prior knowledge identity preservation, so as to improve the visual experience of the audience.

[0005] According to a first aspect of the embodiments of the present disclosure, a method for generating a facial animation based on prior knowledge identity preservation is provided, comprising:

[0006] Extracting features from the source image to obtain source image features, and extracting features from the driving image to obtain driving image features;

[0007] Calculate the background optical flow of the target image based on the background of the source image and the driving image;

[0008] Obtaining the motion optical flow of the target image based on the source image feature, the driving image feature and the background optical flow;

[0009] In response to inputting the source image and the motion optical flow into an identity enhancement generator network, the target image is obtained.

[0010] A second aspect of the disclosed embodiment provides a facial animation generation device based on prior knowledge identity preservation, a feature extraction module for extracting features from a source image to obtain source image features, and extracting features from a driving image to obtain driving image features;

[0011] A background optical flow calculation module, used for calculating the background optical flow of the target image based on the background of the source image and the driving image;

[0012] A motion optical flow calculation module, used for obtaining the motion optical flow of the target image based on the source image features, the driving image features and the background optical flow;

[0013] An image generation module is used to obtain the target image in response to inputting the source image and the motion optical flow into an identity enhancement generator network.

[0014] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, the steps of the above-mentioned method for generating facial animation based on prior knowledge identity preservation are implemented.

[0015] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned method for generating facial animation based on prior knowledge identity preservation are implemented.

[0016] The beneficial effect of the method and device for generating facial animation based on prior knowledge identity preservation provided by the disclosed embodiment is that the disclosed embodiment effectively integrates the character identity information and dynamic expression changes by accurately extracting the features of the source image and the driving image and combining the background optical flow analysis. It not only ensures the consistency and continuity of the character identity in the animation, but also makes the animation effect more coordinated and natural with the background environment through the introduction of the background optical flow. Finally, through the fine processing of the identity enhancement generator network, a facial animation image that not only maintains the identity characteristics but also has dynamic expressiveness is generated, which greatly improves the realism and visual experience of the facial animation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 A schematic diagram of a flow chart of a method for generating facial animation based on prior knowledge identity preservation provided by an embodiment of the present disclosure;

[0019] Figure 2 A schematic diagram of a flow chart of a method for generating facial animation based on prior knowledge identity preservation provided by another embodiment of the present disclosure;

[0020] Figure 3 A flowchart of a method for generating facial animation based on prior knowledge identity preservation provided by another embodiment of the present disclosure;

[0021] Figure 4 This is a rendering of a method for generating facial animation based on prior knowledge identity preservation provided by an embodiment of the present disclosure;

[0022] Figure 5 A comparison diagram of the effects of a method for generating facial animation based on prior knowledge identity preservation provided by an embodiment of the present disclosure;

[0023] Figure 6 A structural block diagram of a facial animation generation device based on prior knowledge identity preservation provided by an embodiment of the present disclosure;

[0024] Figure 7 A schematic block diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.

[0026] In order to make the purpose, technical solutions and advantages of the present disclosure more clear, specific embodiments will be described below in conjunction with the accompanying drawings.

[0027] Please refer to Figure 1 , Figure 1 A flowchart of a method for generating a facial animation based on prior knowledge identity preservation provided by an embodiment of the present disclosure, the method comprising:

[0028] S101: extracting features from a source image to obtain source image features, and extracting features from a driving image to obtain driving image features.

[0029] In this embodiment, the source image refers to the original image used as the basis in the process of generating facial animation, and may include facial features and expressions of the character, etc. The driving image is used to provide the source image with motion information required for animation generation.

[0030] In this embodiment, feature extraction can be performed on the source image through a convolutional neural network in deep learning to obtain source image features, and feature extraction can be performed on the driving image to obtain driving image features.

[0031] For example, the convolution layer in a convolutional neural network performs convolution operations on the source image using different convolution kernels. The convolution kernel slides on the source image and performs dot product operations with the local area of ​​the source image to extract the local features of the source image. For example, for a face image, the shape, texture and other features of the eyes, mouth, nose and other parts can be extracted.

[0032] After the convolution operation, you can use the activation function to perform nonlinear transformation on the result to increase the network's expressiveness. Common activation functions include ReLU (Rectified Linear Unit) and others.

[0033] In order to reduce the dimension of features and reduce the amount of calculation, pooling operations can be performed. Pooling can downsample feature maps. For example, maximum pooling can select the maximum value in a local area as the representative value of the area.

[0034] Repeat the above process: through the combination of multiple convolutional layers, activation functions and pooling layers, higher-level source image features are gradually extracted.

[0035] In this embodiment, the source image features can be used to maintain the identity information of the person, and the driving image features can be used to guide the animation effect of the face.

[0036] S102: Calculate the background optical flow of the target image based on the background of the source image and the driving image.

[0037] In this embodiment, optical flow is a vector field that describes the movement of pixels in an image at different times or between different images, and reflects the movement of an object or scene in the image, including the direction and speed of the movement.

[0038] In this embodiment, the backgrounds of the source image and the driving image are extracted, and the background optical flow of the target image is calculated based on the backgrounds of the source image and the driving image.

[0039] First, the background part is extracted from the source image and the driving image. This embodiment can be implemented by a variety of methods, such as separating the foreground (main object such as human face) and the background in the image based on image segmentation technology; or using a specific algorithm to detect the edges and regions in the image to determine the range of the background.

[0040] Then, the background optical flow of the target image can be calculated by gradient-based methods, block matching-based methods, and energy-based methods. For example, taking the gradient-based method as an example, the gradient-based method uses the change of the brightness of the pixels in the image over time to calculate the optical flow. Assume that the brightness of the pixels in the image remains unchanged in a short time, that is, the optical flow constraint equation is satisfied. By solving this equation, the movement speed of the pixel, that is, the optical flow, can be obtained.

[0041] In this embodiment, the optical flow is processed to obtain the target image background optical flow.

[0042] In this embodiment, the calculated initial optical flow may contain noise and inaccuracy, so it needs to be optimized and adjusted. Filtering techniques, such as Gaussian filtering, median filtering, etc., can be used to remove noise in the optical flow. At the same time, the optical flow can also be adjusted according to some prior knowledge and constraints. For example, if the movement of the known background is smooth, the optical flow can be smoothed to make it more in line with the actual situation. After processing the optical flow, the background optical flow of the target image is obtained, so that the background of the target image can transition and change naturally.

[0043] Exemplary, reference Figure 2 , the source image S and the driving image D are input into the background motion estimator, which is used to estimate the simple animation motion generated by the image background. The background motion estimator can automatically detect the background of the source image S and the driving image D and calculate the affine transformation between the backgrounds as the background optical flow T of the target image bg .

[0044] In this embodiment, the background optical flow of the target image is obtained by analyzing and calculating the background of the source image and the driving image, which provides important background motion information for generating high-quality facial animation.

[0045] S103: Obtaining the motion optical flow of the target image based on the source image features, the driving image features and the background optical flow;

[0046] In this embodiment, by comparing the source image features and the driving image features, the changes in the face between different images can be determined. For example, if the face in the driving image has a specific expression change or head movement, the difference with the source image features can reflect the direction and amplitude of these movements. The background optical flow is merged with the motion trend obtained based on the facial features. For example, the facial motion trend can be adjusted according to the direction and amplitude of the background optical flow to ensure the coordination of the facial motion and the background motion. At the same time, the background optical flow can also provide some constraints for the motion optical flow to avoid unreasonable conflicts between the facial motion and the background motion. The motion optical flow of the target image can be calculated by the target model.

[0047] In this embodiment, the motion optical flow of the target image is calculated by integrating the source image features, the driving image features and the background optical flow information, thereby providing accurate motion guidance for subsequent facial animation generation.

[0048] S104: In response to inputting the source image and the motion optical flow into the identity enhancement generator network, a target image is obtained.

[0049] In this embodiment, the identity enhancement generator network can be a deep learning network, which can be composed of multiple convolutional layers, deconvolutional layers, activation functions, etc.

[0050] The identity enhancement generator network can effectively process image data and combine the information of motion optical flow to generate the target image. For example, the network may have an encoder and decoder structure, where the encoder part is used to extract the features of the source image and the decoder part is used to generate the target image based on these features and motion optical flow.

[0051] The identity enhancement generator network first extracts features from the source image and extracts image features at different levels, which can contain information such as the facial structure, texture, and color of the character. The motion optical flow is fused with the extracted source image features. For example, weighted summation of features and optical flow, use of attention mechanisms, distortion, and mask completion are performed in a specific layer of the network, so that the network can adjust the source image features according to the information of the optical flow. Based on the fused features, the identity enhancement generator network gradually generates the target image through the decoder. The decoder maps the features back to the image space and gradually restores the resolution and details of the image through operations such as deconvolution.

[0052] In this embodiment, after being processed and optimized by the identity enhancement generator network, a target image with a facial animation effect is finally generated. While presenting the facial motion changes in the driving image, the target image maintains the identity features of the person in the source image, making the facial animation more natural and realistic.

[0053] From the above, it can be concluded that this embodiment effectively integrates character identity information and dynamic expression changes by accurately extracting the features of the source image and the driving image and combining background optical flow analysis. It not only ensures the consistency and continuity of the character identity in the animation, but also makes the animation effect more coordinated and natural with the background environment through the introduction of background optical flow. Finally, through the fine processing of the identity enhancement generator network, a facial animation image that maintains identity characteristics and is dynamic and expressive is generated, which greatly improves the realism and visual experience of facial animation.

[0054] In one embodiment of the present disclosure, extracting features from a source image to obtain source image features, and extracting features from a drive image to obtain drive image features, include:

[0055] Preprocessing the source image to obtain a first source image, and preprocessing the drive image to obtain a first drive image;

[0056] Extracting features from the first source image to obtain source image features;

[0057] Feature extraction is performed on the first driving image to obtain driving image features.

[0058] In this embodiment, the first source image is an image obtained by preprocessing the original source image; and the first driving image is an image obtained by preprocessing the original driving image.

[0059] In this embodiment, the source image and the driving image are preprocessed to make the image data more regular and consistent so that the subsequent feature extraction and processing process can be performed more efficiently and accurately. Preprocessing can remove noise in the image, adjust the size and resolution of the image, enhance the contrast of the image, etc. The first source image and the first driving image obtained by the preprocessing of the source image and the driving image are subjected to feature extraction operations respectively to obtain representations that can represent the essential features of the source image and the driving image.

[0060] In this embodiment, feature extraction can use methods such as convolutional neural networks in deep learning. Convolutional neural networks automatically learn features in images, from low-level edge and texture features to high-level facial structure and expression features, through a combination of multiple convolutional layers, pooling layers, and activation functions, to improve image quality and processability.

[0061] It can be concluded from the above that this embodiment preprocesses the source image and the driving image to regularize the image data and improve the image quality, and then extracts features from the preprocessed image. This effectively improves the accuracy and efficiency of subsequent feature extraction and ensures that the extracted image features can more accurately reflect the essential attributes of the source image and the driving image.

[0062] In one embodiment of the present disclosure, preprocessing is performed based on a source image to obtain a first source image, and preprocessing is performed based on a drive image to obtain a first drive image, including:

[0063] Processing the source image through a priori knowledge extraction network to obtain a first face segmentation image and a first face grid image;

[0064] The driving image is processed by a priori knowledge extraction network to obtain a second face segmentation image and a second face grid image;

[0065] Connecting the first face segmentation image and the first face grid image to obtain a first source image;

[0066] The second face segmentation image and the second face grid image are connected to obtain a first driving image.

[0067] In this embodiment, the first face segmentation image is the face segmentation result obtained after processing the source image, and the second face segmentation image is the face segmentation result obtained after processing the driving image; the first face grid image is the face grid representation extracted from the source image, and the second face grid image is the face grid representation extracted from the driving image.

[0068] In this embodiment, the driving image and the source image are preprocessed, and the face can be accurately located through the open source face recognition and face verification library openface, and the driving image and the source image are cropped to a size of 256×256, and then processed frame by frame according to the driving image.

[0069] Exemplary, reference Figure 2 , the source image S and the driving image D are processed by Mediapipe to obtain the corresponding first face segmentation image S m , the second face segmentation image D m and the first face mesh image S l , the second face grid image D l As prior knowledge to guide model training. Then the first face segmentation image S m , the second face segmentation image D m , the first face grid image S l , the second face grid image D l Connect them in channels to get the first source image S of 4 channels in and the first drive image D in , as source faces, driving faces and prior knowledge respectively. Among them, Mediapipe is an open source framework for building, testing and deploying complex multimodal and multi-task machine learning models.

[0070] The first source image S in and the first drive image D in Input to the key point detector, which is used to extract the first source image S in and the first drive image D in The posture key points in different portraits are respectively detected by a ResNet50 network in an unsupervised manner for two sets of key points K in two faces. S and K D , where each group has 50 key points; among them, the key point K S Represents the source image features, key points K D Represents the driving image features.

[0071] From the above, it can be concluded that this embodiment pre-processes the source image and the driving image by introducing a priori knowledge extraction network, generates a face segmentation image and a face grid image, and combines these prior knowledge with the original image to form a first source image and a first driving image, thereby improving the accuracy and efficiency of face image processing. This not only helps the model to more accurately capture the geometric structure and texture information of the face, but also enhances the robustness and generalization ability of the model in complex scenes, and provides richer and more accurate input data for subsequent tasks such as image processing, face synthesis or expression transfer.

[0072] In one embodiment of the present disclosure, in response to inputting a source image and a motion optical flow into an identity enhancement generator network, a target image is obtained, including:

[0073] Encoding based on the source image to obtain a second source image;

[0074] warping and masking the second source image based on motion optical flow to obtain a third source image;

[0075] Decoding is performed based on the third source image to obtain a target image.

[0076] In this embodiment, the second source image is an image representation obtained by encoding the original source image; the third source image is an image obtained after distortion and masking operations of motion optical flow on the basis of the second source image.

[0077] In this embodiment, the purpose of encoding the source image is to convert it from the original image space into a more compact and abstract feature representation. This process can extract key information from the source image, reduce the data dimension, and retain enough features for subsequent processing.

[0078] In this embodiment, the source image can be encoded by an encoder to obtain a second source image; the encoder can be part of a convolutional neural network, and through a combination of convolutional layers, pooling layers and nonlinear activation functions, features of different levels of the source image are gradually extracted.

[0079] In one embodiment of the present disclosure, the second source image is distorted and masked based on the motion optical flow to obtain a third source image, including:

[0080] offsetting each pixel in the second source image based on the motion optical flow to obtain a distorted image;

[0081] A mask operation is performed on the distorted image to obtain a third source image.

[0082] In this embodiment, for each pixel in the second source image, its position is offset according to the corresponding vector in the motion optical flow, thereby realizing the distortion of the image. In this way, the deformation of the face in the source image in the direction indicated by the motion optical flow can be simulated to adapt to the motion of the face in the driving image. The mask can be a binary image or a weight image, which is used to perform pixel-by-pixel multiplication or logical operation on the distorted image after distortion. For example, a mask can be used to remove unreasonable pixel values ​​that may appear in the distortion process, or to highlight specific areas of the face, such as eyes, mouth, etc., to enhance the effect of facial animation. After distortion and masking operations, the third source image is obtained.

[0083] Decoding the third source image is to transform it from the distorted and adjusted feature representation back to the image space to generate the final target image. The decoding process can be implemented using a decoder structure corresponding to the encoding process.

[0084] For example, two sets of key points K S and K D and background optical flow T bg The motion flow estimator is used to estimate the motion representation between the source image and the driving image to predict the overall motion optical flow of the target image. The 50 key points are divided into ten groups of 5 key points, and the key points K are calculated by TPS transformation. S (Source image features) to key points K D The motion estimator integrates ten sets of key point transformations and a set of background affine transformations to obtain the final motion optical flow from the source image to the driving image. At the same time, the motion estimator also generates motion masks of different scale layers to complete the face occlusion.

[0085] The source image S is input into the identity enhancement generator network, an encoder is used to encode the source image, and then the motion optical flow is used to distort and mask the second source image, and then the decoder is used to decode the target image to generate the generated image O. Figure 4 It is the effect diagram of source image, driving image and generated image.

[0086] refer to Figure 3 ,Each layer of the decoder has the same structure. Each layer undergoes a convolution and then two identity embeddings of IADE Resblock. Finally, an upsampling is performed to obtain result images of different sizes. IADEResblock ,adaptively embeds the identity information into the distorted image after two IADE embedding, convolution and normalization operations.

[0087] Among them, the IADE Resblock layer mainly realizes adaptive identity feature embedding. First, the image feature h in After normalization, we get the feature h IN , identity feature Z id After two MLP layers, the weighted parameters are obtained. and , feature h IN With weighting parameters and Perform weighted calculation to obtain the new image feature h id ; Image feature h in After convolution and Sigmoid (function) again, the image feature mask is obtained ,h IN and hid According to the mask Perform weighted calculation to obtain the final output feature h out , to achieve adaptive embedding of identity information, that is

[0088]

[0089] From the above, it can be concluded that this embodiment achieves efficient and precise face animation generation by combining source image encoding, image distortion and mask processing based on motion optical flow, and decoding process. Not only does it retain the identity features of the source image, but it also accurately simulates the dynamic changes of the face through motion optical flow, and uses mask technology to optimize image details, such as processing occluded areas, thereby generating a target image that is both natural and in line with the laws of driving image motion.

[0090] In one embodiment of the present disclosure, it also includes:

[0091] Decoding is performed based on the third source image and the first face segmentation image features to obtain a target image.

[0092] In this embodiment, the first face segmentation image feature is adaptively injected into the image through a decoder to ensure that the source identity is not lost.

[0093] In one embodiment of the present disclosure, decoding is performed based on the third source image and the first face segmentation image features to obtain a target image, including:

[0094] The source image is processed through a priori knowledge extraction network to obtain a first face segmentation image;

[0095] Extracting features from the first face segmentation image to obtain first face segmentation image features;

[0096] Decoding is performed based on the third source image and the first face segmentation image features to obtain a target image.

[0097] In this embodiment, the source image is processed through a priori knowledge extraction network to obtain a first face segmentation image, and then feature extraction is performed on the first face segmentation image. A convolutional neural network or other feature extraction algorithm can be used to extract features containing information such as facial structure, shape, and position.

[0098] The first face segmentation image feature provides information about the division of the face area and the background area and the structure of the face. By fusing these two features, the decoder can take into account both the deformation of the source image and the identity information of the face when generating the target image. The decoder can achieve feature fusion in many ways, such as splicing the two features together at the input layer, or dynamically fusing the two features as needed in different decoding layers.

[0099] For example, if the decoder detects that certain changes in the third source image may cause the source identity to be lost, it can adjust based on the face segmentation image features to emphasize important features of the face, such as the shape and position of the eyes, mouth, nose, etc., to maintain the identity characteristics of the person in the source image.

[0100] It can be concluded from the above that this embodiment can effectively ensure that the source identity is not lost by adaptively injecting the first face segmentation image features into the decoding process. The decoder can adjust the generation process according to the face segmentation image features to ensure that the key features of the face are retained.

[0101] In one embodiment of the present disclosure, it also includes:

[0102] Multiple target images are stitched together to obtain a target image video.

[0103] In this embodiment, in order to ensure that the spliced ​​video has the correct time sequence and coherence, the arrangement order of the target images needs to be determined, which can be determined according to the generation time of the image, a specific number or the corresponding relationship with the driving image.

[0104] For example, if the target images are generated in time order, the images can be sorted according to the timestamp; if they correspond to the frames of the driving images, the order of the target images can be determined according to the playback order of the driving images.

[0105] During the stitching process, you need to set the video parameters, such as frame rate, encoding format, resolution, etc. For each target image, add it as a frame of the video. Add the images in a certain order so that the video can continuously show the changes of the face animation.

[0106] It can be concluded from the above that this embodiment improves the display effect and practicality of face animation by introducing the function of splicing multiple target images to generate a target image video. It not only ensures the time sequence and coherence of the video content, but also allows users to optimize the viewing experience by adjusting video parameters.

[0107] In this embodiment, the evaluation can be performed through a loss function.

[0108] 1. Pyramid-aware loss: A pre-trained VGG-19 network can be used to extract high-quality features of multi-resolution input images, and the distance between the feature maps of the driving image and the generated image at different intermediate layers is calculated to supervise the reconstruction effect at different resolutions. Loss can be expressed as:

[0109]

[0110] in, Represented as the i-th layer of the VGG-19 network, Represented as a driving image, Represented as generated image.

[0111] 2. Equivariance loss: Calculate the equivariance loss to constrain the keypoint detector:

[0112]

[0113] in, is a random TPS transformation, is the key point detector, is the first source image.

[0114] Warp loss: Use the warp loss to further constrain the distorted encoded image in the generator network:

[0115]

[0116] in, Denoted as the i-th layer in the encoder, It is a warping operation performed based on the predicted optical flow Ms→d.

[0117] Identity constraint loss: Introducing identity constraint loss, by calculating the first face segmentation image and generate segmented images The identity information dissimilarity is used to constrain the preservation of identity features:

[0118]

[0119] In this embodiment, the Manhattan distance (L1) can be calculated to represent the reconstruction ability of the model; the Peak Signal-to-Noise Ratio (PSNR) and the Structural Similarity Index Measure (SSIM) are used to evaluate the image generation quality; the Average Euclidean Distance (AEID) is used to measure the degree of preservation of identity information; the Frechet Inception Distance (FID) and the Learned Perceptual Image Patch Similarity (LPIPS) are used to evaluate the similarity between the generated video and the ground-truth video.

[0120] This embodiment can be trained and tested on the VoxCeleb dataset, which contains 22,496 face videos extracted from YouTube. During preprocessing, each video is cropped and resized to 256×256, resulting in a total of 18,025 training videos and 473 test videos. The length of the videos ranges from 64 to 1024 frames.

[0121] like Figure 5 As shown, in order to verify the effectiveness of each module of the model, this embodiment and 4 different models are used to conduct experiments in the above example data set to compare the generated results, which proves the effectiveness of the model.

[0122] Table 1 is a comparison of the results of this embodiment with other methods, including X2Face and FOMM. It can be seen from the table that this method is superior to other methods in most evaluation indicators.

[0123] Table 1 Quantitative results of self-reconstruction and cross-reconstruction. −c indicates cross-reconstruction. ‘↓’ indicates that the smaller the value, the better, and ‘↑’ indicates that the larger the value, the better. The best results are highlighted in bold.

[0124]

[0125] Corresponding to the method for generating facial animation based on prior knowledge identity preservation in the above embodiment, Figure 6 This is a structural block diagram of a facial animation generation device based on prior knowledge identity preservation provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 6 The facial animation generation device 20 based on prior knowledge identity preservation includes: a feature extraction module 21, a background optical flow calculation module 22, a motion optical flow calculation module 23, and an image generation module 24.

[0126] The feature extraction module 21 is used to extract features from the source image to obtain source image features, and to extract features from the drive image to obtain drive image features;

[0127] A background optical flow calculation module 22, used for calculating the background optical flow of the target image based on the background of the source image and the driving image;

[0128] A motion optical flow calculation module 23, used to obtain the motion optical flow of the target image based on the source image features, the driving image features and the background optical flow;

[0129] The image generation module 24 is used to obtain a target image in response to inputting the source image and the motion optical flow into the identity enhancement generator network.

[0130] In one embodiment of the present disclosure, the feature extraction module 21 is specifically used for:

[0131] Preprocessing the source image to obtain a first source image, and preprocessing the drive image to obtain a first drive image;

[0132] Extracting features from the first source image to obtain source image features;

[0133] Feature extraction is performed on the first driving image to obtain driving image features.

[0134] In one embodiment of the present disclosure, the feature extraction module 21 is further used to:

[0135] Processing the source image through a priori knowledge extraction network to obtain a first face segmentation image and a first face grid image;

[0136] The driving image is processed by a priori knowledge extraction network to obtain a second face segmentation image and a second face grid image;

[0137] Connecting the first face segmentation image and the first face grid image to obtain a first source image;

[0138] The second face segmentation image and the second face grid image are connected to obtain a first driving image.

[0139] In one embodiment of the present disclosure, the image generation module 24 is specifically used to:

[0140] Encoding based on the source image to obtain a second source image;

[0141] warping and masking the second source image based on motion optical flow to obtain a third source image;

[0142] Decoding is performed based on the third source image to obtain a target image.

[0143] In one embodiment of the present disclosure, the image generation module 24 is further configured to:

[0144] Decoding is performed based on the third source image and the first face segmentation image features to obtain a target image.

[0145] In one embodiment of the present disclosure, the image generation module 24 is further configured to:

[0146] The source image is processed through a priori knowledge extraction network to obtain a first face segmentation image;

[0147] Extracting features from the first face segmentation image to obtain first face segmentation image features;

[0148] Decoding is performed based on the third source image and the first face segmentation image features to obtain a target image.

[0149] In one embodiment of the present disclosure, the facial animation generation device 20 based on prior knowledge identity preservation further includes: an image video generation module; the image video generation module is specifically used to:

[0150] Multiple target images are stitched together to obtain a target image video.

[0151] See also Figure 7 , Figure 7 A schematic block diagram of an electronic device provided by an embodiment of the present disclosure. Figure 7 The electronic device 300 in the embodiment shown may include: one or more processors 301, one or more input devices 302, one or more output devices 303 and one or more memories 304. The processors 301, input devices 302, output devices 303 and memories 304 communicate with each other via a communication bus 305. The memory 304 is used to store computer programs, which include program instructions. The processor 301 is used to execute the program instructions stored in the memory 304. The processor 301 is configured to call the program instructions to execute the functions of the modules in the above-mentioned device embodiments, such as Figure 6 The functions of modules 21 to 24 are shown.

[0152] It should be understood that in the embodiment of the present disclosure, the processor 301 may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0153] The input device 302 may include a touch panel, a fingerprint collection sensor (for collecting the user's fingerprint information and fingerprint direction information), a microphone, etc., and the output device 303 may include a display (LCD, etc.), a speaker, etc.

[0154] The memory 304 may include a read-only memory and a random access memory, and provide instructions and data to the processor 301. A portion of the memory 304 may also include a non-volatile random access memory. For example, the memory 304 may also store information about the device type.

[0155] In a specific implementation, the processor 301, input device 302, and output device 303 described in the embodiments of the present disclosure can execute the implementation methods described in the first and second embodiments of the facial animation generation method based on prior knowledge identity preservation provided in the embodiments of the present disclosure, and can also execute the implementation methods of the electronic device described in the embodiments of the present disclosure, which will not be repeated here.

[0156] In another embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by the processor, all or part of the processes in the above-mentioned embodiment method are implemented, and the computer program can also be completed by instructing the relevant hardware through the computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0157] The computer-readable storage medium may be an internal storage unit of the electronic device of any of the aforementioned embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of the electronic device and an external storage device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0158] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0159] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the electronic devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0160] In the several embodiments provided in the present application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces or units, or it can be an electrical, mechanical or other form of connection.

[0161] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present disclosure.

[0162] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0163] The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present disclosure, and these modifications or replacements should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A method for generating facial animation based on prior knowledge identity preservation, characterized in that: include: Extracting features from the source image to obtain source image features, and extracting features from the driving image to obtain driving image features; Calculate the background optical flow of the target image based on the background of the source image and the driving image; Obtaining the motion optical flow of the target image based on the source image feature, the driving image feature and the background optical flow; In response to inputting the source image and the motion optical flow into an identity enhancement generator network, obtaining the target image; The step of extracting features from the source image to obtain source image features and extracting features from the drive image to obtain drive image features comprises: Preprocessing the source image to obtain a first source image, and preprocessing the drive image to obtain a first drive image; Extracting features from the first source image to obtain features of the source image; Extracting features from the first driving image to obtain the driving image features; The preprocessing based on the source image to obtain the first source image and the preprocessing based on the drive image to obtain the first drive image include: Processing the source image through a priori knowledge extraction network to obtain a first face segmentation image and a first face grid image; Processing the driving image through a priori knowledge extraction network to obtain a second face segmentation image and a second face grid image; Connecting the first face segmentation image and the first face grid image to obtain a first source image; Connecting the second face segmentation image and the second face grid image to obtain a first drive image; The step of obtaining the target image in response to inputting the source image and the motion optical flow into an identity enhancement generator network comprises: Perform encoding based on the source image to obtain a second source image; warping and masking the second source image based on the motion optical flow to obtain a third source image; Decoding based on the third source image to obtain the target image; Also includes: Decoding is performed based on the third source image and the first face segmentation image features to obtain the target image; The decoding based on the third source image and the first face segmentation image features to obtain the target image includes: Processing the source image through a priori knowledge extraction network to obtain a first face segmentation image; Performing feature extraction on the first face segmentation image to obtain first face segmentation image features; Decoding is performed based on the features of the third source image and the first face segmentation image to obtain the target image.

2. The method for generating facial animation based on prior knowledge identity preservation as claimed in claim 1, characterized in that: Also includes: A plurality of the target images are spliced ​​together to obtain a target image video.

3. A facial animation generation device based on prior knowledge identity preservation, characterized in that: include: A feature extraction module is used to extract features from a source image to obtain source image features, and to extract features from a drive image to obtain drive image features; A background optical flow calculation module, used for calculating the background optical flow of the target image based on the background of the source image and the driving image; A motion optical flow calculation module, used for obtaining the motion optical flow of the target image based on the source image features, the driving image features and the background optical flow; An image generation module, configured to obtain the target image in response to inputting the source image and the motion optical flow into an identity enhancement generator network; The step of extracting features from the source image to obtain source image features and extracting features from the drive image to obtain drive image features comprises: Preprocessing the source image to obtain a first source image, and preprocessing the drive image to obtain a first drive image; Extracting features from the first source image to obtain features of the source image; Extracting features from the first driving image to obtain the driving image features; The preprocessing based on the source image to obtain the first source image and the preprocessing based on the drive image to obtain the first drive image include: Processing the source image through a priori knowledge extraction network to obtain a first face segmentation image and a first face grid image; Processing the driving image through a priori knowledge extraction network to obtain a second face segmentation image and a second face grid image; Connecting the first face segmentation image and the first face grid image to obtain a first source image; Connecting the second face segmentation image and the second face grid image to obtain a first drive image; The step of obtaining the target image in response to inputting the source image and the motion optical flow into an identity enhancement generator network comprises: Perform encoding based on the source image to obtain a second source image; warping and masking the second source image based on the motion optical flow to obtain a third source image; Decoding based on the third source image to obtain the target image; Also includes: Decoding is performed based on the third source image and the first face segmentation image features to obtain the target image; The decoding based on the third source image and the first face segmentation image features to obtain the target image includes: Processing the source image through a priori knowledge extraction network to obtain a first face segmentation image; Performing feature extraction on the first face segmentation image to obtain first face segmentation image features; Decoding is performed based on the features of the third source image and the first face segmentation image to obtain the target image.

4. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 2 are implemented.

5. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 2 are implemented.

Images