Character expression driving method and device, storage medium and computer equipment

Abstract

According to the character expression driving method and device, the storage medium and the computer equipment provided by the invention, during expression driving, the continuous video frame images and the expression driving target in the character video are determined firstly, so that expression driving is realized on the video frame images, and the naturalness of an expression driving effect is ensured; and then face alignment is carried out based on the face key points in the video frame image to obtain a standard face image, expression driving is carried out on the standard face image based on the expression driving target through the target expression driving model, an expression driving video is generated, and thus the quality of the expression driving effect is improved. The target expression driving model is formed by the three-dimensional convolutional neural network and is used for mapping the two-dimensional image to the three-dimensional hidden space for feature coding and expression driving, so that the model can accurately track human faces at various angles in the video frame image and perform expression editing and driving; and expression natural connection of video frame images between frames is realized under the condition that the quality of a figure video is not influenced.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, storage medium and computer equipment for driving human expression. Background Art

[0002] With the widespread application of AIGC (Artificial Intelligence Generated Content) technology in the video field, especially in high-cost productions such as movies, TV series, short dramas and animations, the demand for driving and editing video character expressions is growing. This demand aims to achieve the editing of character expressions in a more flexible and fast way, thereby effectively reducing shooting costs, shortening production cycles, and improving the team's efficiency in video content creation.

[0003] The existing human expression driving methods are highly dependent on the training of a large amount of data, which has a high limitation on the scope of application and is not ideal in high-quality scenes. Although the 3D digital human expression driving method performs well in 3D modeling scenes, it is not suitable for direct application in human video driving due to its technical characteristics. Therefore, the current human expression driving method has high limitations and cannot meet the needs of video expression driving at the film and television level. Summary of the invention

[0004] The purpose of this application is to solve at least one of the above-mentioned technical defects, especially the technical defect that the character expression driving method in the prior art has high limitations and cannot meet the film and television level video expression driving requirements.

[0005] The present application provides a method for driving a character expression, the method comprising:

[0006] Acquire a person video containing a human face and an expression driving target of the person video, and parse to obtain continuous video frame images in the person video;

[0007] Using a target face detection model to identify key face points in the video frame image, and performing face alignment on the video frame image based on the key face points to obtain a standard face image;

[0008] Determine a target expression driving model; the target expression driving model is composed of a three-dimensional convolutional neural network, which is used to map a two-dimensional image into a three-dimensional latent space for feature encoding and expression driving;

[0009] The standard face image and the expression driving target are input into the target expression driving model, and the target expression driving model is used to perform expression driving on the standard face image based on the expression driving target to output an expression driving video.

[0010] Optionally, the identifying and obtaining facial key points in the video frame image using a target face detection model includes:

[0011] Determine a target face detection model, wherein the target face detection model includes a face recognition network and a key point detection network;

[0012] Performing face recognition on the video frame image using the face recognition network to obtain a face position frame in the video frame image;

[0013] In the key point detection network, key point detection is performed on the original face image based on the face position frame to obtain face key points.

[0014] Optionally, the training process of the target face detection model includes:

[0015] Acquire a sample face image marked with a real face position frame, wherein the real face position frame is marked with real face key points;

[0016] Inputting the sample face image into a preset initial face detection model, obtaining a predicted face position frame output by the initial face detection model for the sample face image, and a predicted confidence level and predicted face key points of the predicted face position frame;

[0017] Training the initial face detection model with the goal of making the predicted face position frame and the predicted face key points approach the real face position frame and the real face key points respectively;

[0018] When the initial face detection model meets the preset training conditions, the trained initial face detection model is lightweighted to obtain a target face detection model.

[0019] Optionally, the training of the initial face detection model with the goal of making the predicted face position frame and the predicted face key points approach the real face position frame and the real face key points respectively includes:

[0020] Determine a position loss value of the predicted face position frame based on the real face position frame, and calculate a confidence loss value of the prediction confidence according to the position loss value;

[0021] Determine a key point loss value of the predicted face key point based on the real face key point;

[0022] Update the parameters in the initial face detection model according to the position loss value, the confidence loss value and the key point loss value.

[0023] Optionally, performing face alignment on the video frame images based on the face key points to obtain a standard face image includes:

[0024] The face region in the video frame image is located according to the face key points, and the located face region is aligned to a preset standard position using affine transformation technology to form a standard face image.

[0025] Optionally, the target expression driving model includes a latent space mapping module, a key point generation module, a spatial local deformation module and a feature decoding module;

[0026] The step of using the target expression driving model to perform expression driving on the standard face image based on the expression driving target includes:

[0027] The latent space mapping module is used to extract low-order and high-frequency features of the standard face image, and the low-order and high-frequency features are latently encoded and feature decoupled to generate a three-dimensional image corresponding to the standard face image in the latent space;

[0028] Determining a plurality of deformation regions in the three-dimensional image and a deformation feature corresponding to each deformation region by means of the key point generation module;

[0029] Determine the deformation area and deformation direction corresponding to the expression-driven target by using the spatial local deformation module, and perform feature editing on the deformation features of the deformation area based on the deformation direction to generate an expression-driven image;

[0030] The expression-driven image is input into the feature decoding module, so that the feature decoding module uses a generator composed of a SPADE network to upsample and decode the expression-driven image, so as to map the expression-driven image to a two-dimensional space and generate an expression-driven video.

[0031] Optionally, each deformation region is composed of a plurality of deformation key points;

[0032] The determining of the plurality of deformation regions in the three-dimensional image and the deformation features corresponding to each deformation region by the key point generation module includes:

[0033] Determine, by means of the key point generation module, a plurality of deformation regions in the three-dimensional image and the three-dimensional coordinates of each deformation key point in each deformation region;

[0034] For each deformation area, Gaussian distribution technology is used in the key point generation module to calculate the influence of the three-dimensional coordinates of each deformation key point in the deformation area, obtain the characteristic influence value of each deformation key point in the three-dimensional image, and perform weighted summation on each characteristic influence value to form the deformation feature of the deformation area.

[0035] The present application also provides a character expression driving device, comprising:

[0036] A data acquisition module is used to acquire a person video containing a human face and an expression driving target of the person video, and parse and obtain a video frame image of the person video;

[0037] A face recognition module, used to identify key points of faces in the video frame image using a target face detection model, and perform face alignment on the video frame image based on the key points of faces to obtain a standard face image;

[0038] A model determination module is used to determine a target expression driving model; the target expression driving model is composed of a three-dimensional convolutional neural network, which is used to map a two-dimensional image into a three-dimensional latent space for feature encoding and expression driving;

[0039] The expression driving module is used to input the standard face image and the expression driving target into the target expression driving model, and use the target expression driving model to perform expression driving on the standard face image based on the expression driving target to output an expression driven video.

[0040] The present application also provides a storage medium, in which computer-readable instructions are stored. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the character expression driving method described in any one of the above embodiments.

[0041] The present application also provides a computer device, comprising: one or more processors, and a memory;

[0042] The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the character expression driving method described in any one of the above embodiments are performed.

[0043] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0044] The character expression driving method, device, storage medium and computer equipment provided by the present application can first determine the expression driving target of the character video and parse the continuous video frame images in the character video when driving the expression of the character in the character video, so that the expression driving can be realized on the video frame image to ensure the naturalness of the expression driving effect; then the target face detection model can be used to identify the key points of the face in the video frame image, and the video frame image can be aligned based on the key points of the face to obtain a standard face image, so as to drive the standard face image based on the expression driving target through the target expression driving model, and output it as an expression driving video, so as to improve the quality of the expression driving effect. Since the target expression driving model is composed of a three-dimensional convolutional neural network, which is used to map the two-dimensional image into a three-dimensional latent space for feature encoding and expression driving, the target expression driving model can accurately track the face at various angles in the video frame image and perform expression editing and driving, so as to achieve the natural connection of the expression between frames of the video frame image without affecting the quality of the character video. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0046] Figure 1 A flowchart of a method for driving a character expression provided in an embodiment of the present application;

[0047] Figure 2 A schematic diagram of a process for detecting key points of a face provided in an embodiment of the present application;

[0048] Figure 3 A schematic diagram of a process for updating parameters of an initial face detection model provided in an embodiment of the present application;

[0049] Figure 4 A schematic diagram of a process of applying a target expression driven model provided in an embodiment of the present application;

[0050] Figure 5 A schematic diagram of the structure of a character expression driving device provided in an embodiment of the present application;

[0051] Figure 6 A schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0053] The existing human expression driving methods are highly dependent on the training of a large amount of data, which has a high limitation on the scope of application and is not ideal in high-quality scenes. Although the 3D digital human expression driving method performs well in 3D modeling scenes, it is not suitable for direct application in human video driving due to its technical characteristics. Therefore, the current human expression driving method has high limitations and cannot meet the needs of video expression driving at the film and television level.

[0054] Based on this, this application proposes the following technical solutions, see below for details:

[0055] In one embodiment, Figure 1 As shown, Figure 1 A flowchart of a method for driving a character expression provided in an embodiment of the present application; the present application provides a method for driving a character expression, see below for details:

[0056] S110: Acquire a person video containing a face and an expression driving target of the person video, and parse to obtain continuous video frame images in the person video.

[0057] In this step, when the user needs to drive the expression of a film or TV character, the user can upload the character video of the film or TV character to the computer device, and set the expression driving target of the character video in the computer device. The computer device can then parse the character video to obtain continuous video frame images, and then realize expression driving on the video frame images to ensure the naturalness of the expression driving effect.

[0058] It can be understood that the expression-driven goal refers to the facial expression changes or action effects that the user hopes to achieve in the character expression driving process, such as specific expressions such as smile, surprise, frown, etc.; the expression-driven goal here can not only include the action changes of facial features, but also the overall expression dynamic effects, such as the transition from a neutral expression to a smile, thereby achieving high-quality expression-driven effects for film and television characters.

[0059] In addition, the video frame image of the present application refers to a frame-by-frame image sequence after the character video is parsed. To elaborate, the essence of character video is composed of a series of rapidly switched static images (frames). When the frame rate is fast enough, a continuous motion visual effect will be produced. Therefore, when the present application performs expression-driven expression on the character video, it can parse the facial features of each frame image, and then apply the expression-driven target to these frames to ensure that the facial expressions of the characters in the character video can change as expected, thereby avoiding abrupt jumps or incoherent expression effects, thereby keeping the changes in the character's expression smooth and natural.

[0060] Specifically, the computer device can use a video processing tool or library to parse and obtain continuous video frame images in the character video, and then apply the expression driving target set by the user to the video frame images. By adjusting the expression features of the character's face, such as the dynamic changes of the eyes, eyebrows, and lips, the computer can accurately drive the character's expression changes in each frame, ensuring that the transition of these expressions between frames is smooth and natural. Therefore, the present application can achieve a specific expression effect while maintaining the original natural state of the character video.

[0061] S120: using the target face detection model to identify key points of the face in the video frame image, and performing face alignment on the video frame image based on the key points of the face to obtain a standard face image.

[0062] In this step, after obtaining the video frame image of the character video through step S110, the computer device can obtain a preset target face detection model, and then use the target face detection model to identify the facial key points in the video frame image, so that the video frame image can be aligned based on the facial key points to obtain a standard face image. Through the standard face image, the present application can further improve the naturalness of expression driving.

[0063] The target face detection model refers to a model that detects faces on input video frames and identifies key points of faces. Key points of faces here refer to several important points distributed in specific parts of the face, such as eyes, nose, mouth, etc. These points are used to describe the geometric structure and morphological features of the face. Through key points of faces, computer equipment can accurately identify the posture, shape and expression of the face, thereby improving the natural effect of expression-driven.

[0064] Specifically, after identifying the facial key points in the video frame image, the computer device can align the faces in the image according to these facial key points, thereby standardizing the faces in the video frame image, reducing image differences caused by shooting conditions or lighting conditions, and maintaining a uniform scale, ensuring that subtle changes in facial expressions can be better captured in the subsequent expression-driven process, making changes in facial expressions more precise and coherent.

[0065] S130: Determine a target expression driving model; the target expression driving model is composed of a three-dimensional convolutional neural network, which is used to map a two-dimensional image into a three-dimensional latent space for feature encoding and expression driving.

[0066] In this step, after obtaining the standard face image through step S120, the computer device may also determine a target expression driving model to drive the facial expression in the standard face image through the target expression driving model.

[0067] It is understandable that the target expression driving model of the present application is composed of a three-dimensional convolutional neural network, which is used to map a two-dimensional image into a three-dimensional latent space for feature encoding and expression driving. Among them, the three-dimensional convolutional neural network can handle more complex spatial and temporal features compared to the traditional two-dimensional convolutional network. Specifically, the three-dimensional convolutional neural network can not only extract the features of facial expressions in a single-frame two-dimensional image, but also combine the multi-frame information in the video to capture the changes in expression in the time dimension. Therefore, the target expression driving model can map the two-dimensional image into a three-dimensional latent space, thereby performing richer feature encoding of facial expressions. This feature encoding can not only describe the static expression of the face, but also reflect the dynamic changes of the expression, thereby achieving fine expression driving.

[0068] It should be noted that, since the target expression driven model has a strong three-dimensional spatial perception capability, it can accurately track the facial expression changes of the face at different angles and different postures in the video frame image, so that the model can not only process the frontal face, but also can perform high-precision expression capture and editing of the face in a variety of complex postures such as side, top view, and top view. Therefore, in this application, through the learning and reasoning of the target expression driven model, no matter what angle the characters in the character video appear at, it can track the subtle changes in their facial expressions and drive them according to the preset expression driven target.

[0069] In addition, the target expression driven model is not limited to simple expression recognition, but also has the ability to deeply edit and generate facial expressions. Through feature encoding in the three-dimensional latent space, the model can achieve rich expression operations, such as exaggerating a certain expression, smoothing expression transitions, or applying changes in a specific expression to the entire sequence of video frame images, and this process will not affect the overall quality of the video. Therefore, when the model performs expression driving, it can ensure that the expression changes between frames are naturally connected, avoiding abrupt or discontinuous expression transitions, making the facial expressions of people in the character video more realistic and natural.

[0070] S140: Inputting the standard face image and the expression driving target into the target expression driving model, and using the target expression driving model to perform expression driving on the standard face image based on the expression driving target, so as to output an expression driving video.

[0071] In this step, after determining the target expression driving model through step S130, the computer device can input the standard facial image and the expression driving target into the target expression driving model, and use the target expression driving model to perform expression driving on the standard facial image based on the expression driving target to output a high-quality and natural expression driven video.

[0072] Specifically, after receiving the standard face image and the expression-driven target, the target expression-driven model can process and edit the input expression face image frame by frame according to the expression-driven target, and finally output the expression-driven video. Since the model has been trained with a large amount of face data and expression changes, it has highly intelligent feature extraction and transformation capabilities and can accurately capture and manipulate subtle changes in facial expressions. Therefore, when the model edits the facial expressions of the standard face image frame by frame in the time dimension, it can ensure that the natural sense of the character's face is not destroyed, and the expression changes smoothly and in line with the dynamic changes of real facial expressions.

[0073] In the above embodiment, when performing expression driving on a person in a person video, the expression driving target of the person video can be determined first, and the continuous video frame images in the person video can be obtained by parsing, so that expression driving can be realized on the video frame images to ensure the naturalness of the expression driving effect; then the target face detection model can be used to identify the key points of the face in the video frame image, and the video frame image can be aligned based on the key points of the face to obtain a standard face image, so that the standard face image can be expression driven based on the expression driving target through the target expression driving model, and output as an expression driving video, so as to improve the quality of the expression driving effect. Since the target expression driving model is composed of a three-dimensional convolutional neural network, which is used to map a two-dimensional image into a three-dimensional latent space for feature encoding and expression driving, the target expression driving model can accurately track the face at various angles in the video frame image and perform expression editing and driving, so as to achieve the natural connection of the expression between frames of the video frame image without affecting the quality of the person video.

[0074] In one embodiment, Figure 2 As shown, Figure 2 A schematic diagram of a process for detecting key points of a face provided in an embodiment of the present application; Figure 2 In step S120, using the target face detection model to identify key face points in the video frame image may include:

[0075] S121: Determine a target face detection model, where the target face detection model includes a face recognition network and a key point detection network.

[0076] S122: Perform face recognition on the video frame image using a face recognition network to obtain a face position frame in the video frame image.

[0077] S123: Perform key point detection on the original face image based on the face position frame in the key point detection network to obtain face key points.

[0078] In this embodiment, when the computer device performs facial key point recognition on a video frame image, it may first determine a target face detection model, and then use the model to recognize and obtain facial key points in the video frame image.

[0079] Specifically, since the target face detection model includes a face recognition network and a key point detection network, after the computer device inputs the video frame image into the target face detection model, the target face detection model can first perform an in-depth analysis of each frame of the video frame image through the face recognition network to identify whether there is a face in the image. When the face recognition network detects a face, it will generate a face position frame, that is, frame the specific area where the face is located in the frame image, ensuring that the computer device can accurately locate the face part in the video frame for subsequent processing. Then, based on the identified face position frame, the key point detection network will focus on analyzing specific parts of the face, such as eyes, nose, mouth, eyebrows, etc., and identify the geometric positions of the key points of the face to accurately depict the geometric structure and expression characteristics of the person's face.

[0080] Therefore, in the process of identifying facial key points, the present application can first determine the overall position of the face, and then perform key point detection on this basis through close cooperation between the face recognition network and the key point detection network; in this way, the present application can significantly improve the efficiency and accuracy of detection, avoid unnecessary calculations on the entire image, and thereby improve the overall efficiency of character expression driving.

[0081] In one embodiment, the training process of the target face detection model in step S120 may include:

[0082] S124: Acquire a sample face image annotated with a real face position frame, wherein the real face position frame is annotated with real face key points.

[0083] S125: Input the sample face image into a preset initial face detection model to obtain a predicted face position frame output by the initial face detection model for the sample face image, a predicted confidence level of the predicted face position frame, and predicted face key points.

[0084] S126: Training an initial face detection model with the goal of making the predicted face position frame and the predicted face key points approach the real face position frame and the real face key points respectively.

[0085] S127: When the initial face detection model meets the preset training conditions, the trained initial face detection model is lightweighted to obtain a target face detection model.

[0086] In this embodiment, when training the target face detection model, the computer device can first obtain sample face images marked with real face position boxes and real face key points, such as continuous images in character videos in different scenes. The images can be pre-processed images, such as face alignment and other processing, and then the processed sample face images are input into the preset initial face detection model for iterative training, and the trained model is lightweight to obtain the target face detection model. Therefore, the size of the target face detection model can be within 3M and the inference speed is within 10ms.

[0087] Specifically, during the training process, the computer device obtains a preprocessed sample face image, which is annotated with a corresponding sample label, namely, a real face position frame and real face key points in the real face position frame. Therefore, the training sample with the sample label is input into the preset initial face detection model for forward propagation to train the model, and the preset loss function is used to tune the model parameters during the back propagation process of the model. When the model meets certain training conditions or parameter convergence conditions, such as the number of iterations reaches the set value, the training is considered to be completed. At this time, the trained model can be used as the final target face detection model. In addition, the present application can also store the trained target face detection model so that when performing face detection later, the pre-stored target face detection model can be directly called to perform face detection on the video frame image.

[0088] Furthermore, the target face detection model in this application can select Mobilenetv3 (lightweight deep learning network) as the basic model for improvement and training. When improving the network structure of Mobilenetv3, the parameters of channel (number of channels), block (module layer), and linear (linear layer) can be reduced to reduce the amount of calculation and operation time of the model during the entire training and reasoning process, and improve the scope of use of the model by reducing the model's requirements for computing power; and the output parameters of the network can be adjusted to 5, namely the X coordinate of the upper left corner of the face detection frame, the Y coordinate of the upper left corner of the face detection frame, the X coordinate of the lower right corner of the face detection frame, the Y coordinate of the lower right corner of the face detection frame, and the confidence of the face detection frame, so as to supervise the training process of the confidence of the face detection frame.

[0089] In one embodiment, Figure 3 As shown, Figure 3 A schematic diagram of a process for updating parameters of an initial face detection model provided in an embodiment of the present application; Figure 3 In step S126, the initial face detection model is trained with the goal of making the predicted face position frame and the predicted face key points approach the real face position frame and the real face key points respectively, including:

[0090] S1261: Determine a position loss value of the predicted face position frame based on the real face position frame, and calculate a confidence loss value of the prediction confidence based on the position loss value.

[0091] S1262: Determine the key point loss value of the predicted face key points based on the real face key points.

[0092] S1263: Update the parameters in the initial face detection model according to the position loss value, the confidence loss value and the key point loss value.

[0093] In this embodiment, in each iteration process, the computer device can determine the position loss value of the predicted face position frame based on the real face position frame, and then calculate the confidence loss value of the prediction confidence based on the position loss value, and can also determine the key point loss value of the predicted face key point based on the real face key point. Finally, the computer device can update the parameters in the initial face detection model according to the position loss value, the confidence loss value and the key point loss value, so as to complete the training of the model in the current round.

[0094] Specifically, when updating the parameters of the model, the computer device can use the position loss function to calculate the position loss value between the predicted face position frame and the real face position frame. The position loss value reflects the accuracy of the position frame in the current iteration, so the parameters of the model can be further adjusted through the position loss value. After determining the position loss function, the computer device can also use the confidence loss function to calculate the position loss value and the prediction confidence to obtain the confidence loss value of the prediction confidence. The confidence loss value can be used to measure the accuracy of the model in judging whether a certain area does contain a face. It reflects the confidence of the model in its own prediction results. Therefore, by adjusting the parameters of the model through the confidence loss value, the reliability of the model prediction results can be improved.

[0095] At the same time, the computer device can also use the key point loss function to calculate the key point loss value between the predicted face key points and the real face key points. The key point loss value can be used to measure the accuracy of the model in predicting facial details, especially the specific positions of facial features (such as eyes, nose, mouth, etc.). Therefore, the computer device can also use the key point value to further adjust the parameters of the model.

[0096] Furthermore, after calculating the position loss value, confidence loss value, and key point loss value, the computer device can perform comprehensive calculations based on these loss values, and then update the parameters in the model through the backpropagation algorithm. Through backpropagation, the computer device can gradually adjust the various weights and parameters of the model with the goal of reducing the loss value, so that in the next iteration, the model can make more accurate predictions.

[0097] In one embodiment, performing face alignment on the video frame images based on face key points to obtain a standard face image in step S120 may include:

[0098] S128: Locating the face region in the video frame image according to the face key points, and aligning the located face region to a preset standard position using affine transformation technology to form a standard face image.

[0099] In this embodiment, after the computer device identifies the facial key points in the video frame image, it can first locate the facial area in the video frame image according to the facial key points, and then use affine transformation technology to align the located facial area to a preset standard position to form a standard facial image, so that expression driving can be performed on the standard facial image, thereby improving the facial expression driving effect.

[0100] Among them, affine transformation technology refers to a geometric transformation method that can perform linear transformation on an image while keeping the straight lines and proportional relationships in the image unchanged. To elaborate, affine transformation can translate, scale, rotate, shear, and flip an image, and during these transformations, the originally parallel straight lines in the image remain parallel, and the properties such as angles and distances will change accordingly, but the linearity and relative proportional relationships remain unchanged, so that all faces in the image have a consistent structure.

[0101] It is understandable that facial key points usually include the coordinates of facial features such as eyes, nose, and mouth corners. Positioning the facial area through facial key points can provide an accurate reference for subsequent affine transformation. Then, the computer device can use affine transformation technology to align the located facial area with the preset standard position to ensure the consistency between the cross-frame images in the video frame image, thereby improving the accuracy and naturalness of the subsequent expression drive.

[0102] In one embodiment, Figure 4 As shown, Figure 4 A schematic diagram of a process of applying a target expression driven model provided in an embodiment of the present application; Figure 4 In step S140, the target expression driving model includes a latent space mapping module, a key point generation module, a spatial local deformation module and a feature decoding module; wherein the step of using the target expression driving model to drive the standard face image based on the expression driving target may include:

[0103] S141: extracting low-order and high-frequency features of the standard face image using a latent space mapping module, and performing latent space encoding and feature decoupling on the low-order and high-frequency features to generate a three-dimensional image corresponding to the standard face image in the latent space.

[0104] S142: Determine multiple deformation regions in the three-dimensional image and deformation features corresponding to each deformation region through a key point generation module.

[0105] S143: Determine the deformation area and deformation direction corresponding to the expression-driven target by using the spatial local deformation module, and perform feature editing on the deformation features of the deformation area based on the deformation direction to generate an expression-driven image.

[0106] S144: Input the expression-driven image into the feature decoding module, so that the feature decoding module uses a generator composed of a SPADE network to upsample and decode the expression-driven image, so as to map the expression-driven image to a two-dimensional space and generate an expression-driven video.

[0107] In this embodiment, the target expression driving model is composed of multiple modules, namely, a latent space mapping module, a key point generation module, a spatial local deformation module and a feature decoding module. In the process of the target expression driving model performing expression driving on the standard face image, each module performs a corresponding function so that the target expression driving model can output an expression driving video.

[0108] Among them, the latent space mapping module is mainly responsible for extracting the low-order and high-frequency features of the standard face image, and performing latent space encoding and feature decoupling on the low-order and high-frequency features, so as to generate a three-dimensional image corresponding to the standard face image in the latent space; the key point generation module is mainly responsible for determining multiple deformation areas in the three-dimensional image and the deformation features corresponding to each deformation area; the spatial local deformation module is mainly responsible for determining the deformation area and deformation direction corresponding to the expression-driven target, and performs feature editing on the deformation features of the deformation area based on the deformation direction to generate an expression-driven image; the feature decoding module is mainly responsible for upsampling and decoding the expression-driven image using a generator composed of the SPADE network, so as to map the expression-driven image to a two-dimensional space and generate an expression-driven video.

[0109] It can be understood that in the feature decoding module, the generator used for upsampling decoding is constructed by the SPADE network (Spatially-Adaptive Denormalization), which is mainly used for image generation tasks, especially the task of generating images guided by semantic segmentation maps or other spatial structures. The SPADE here is a spatially adaptive normalization method that can dynamically adjust the features of the generated image according to the input semantic mask map. Since the traditional normalization method causes the feature map of each pixel to be normalized in the same way, resulting in the loss of spatial information, the present application can introduce the SPADE network, and introduce different adjustment parameters (scale and offset) according to the semantic category of each pixel during normalization, thereby ensuring that the spatial information is retained during the normalization process, so that the generated image maintains the consistency of details and structure while the generation effect is more realistic and natural.

[0110] Specifically, since the RGB image can be regarded as a kind of low-order high-frequency feature with strong coupling, the low-order high-frequency features of the standard face image can be extracted through the latent space mapping module, and then the low-order high-frequency features are encoded and decoupled in the latent space to deform the face area of ​​the standard face image in the latent space. After mapping the standard face image to the three-dimensional latent space, the key point generation module can determine multiple deformation areas in the latent space, such as the eyes, nose, mouth and other areas, and then generate a group of key points corresponding to each deformation area, and then calculate the deformation features of each deformation area based on each group of key points; therefore, the key point generation module can control the deformation features of each deformation area by controlling each group of key points.

[0111] In addition, in order to support global and local facial expression editing and driving, the spatial local deformation module can change the deformation characteristics of the corresponding deformation area by controlling any one or more groups of key points, and then can realize the expression editing of the corresponding deformation area without affecting the pixels in other areas. For example, the spatial local deformation module can control the opening and closing of the character's mouth through a set of key points. This set of key points cannot control the deformation of the area other than the mouth, thereby realizing the constraint of the expression driving range and ensuring the high quality of the output image. After the spatial local deformation module generates an expression driving image, the feature decoding module can decode the three-dimensional latent spatial features of the image into an RGB image by upsampling decoding, and output it to form an expression driving video as a character video with user-defined expressions.

[0112] In one embodiment, each deformation region in step S143 is composed of a plurality of deformation key points; wherein the step of determining the plurality of deformation regions in the three-dimensional image and the deformation features corresponding to each deformation region by the key point generation module may include:

[0113] S1431: Determine multiple deformation regions in the three-dimensional image and the three-dimensional coordinates of each deformation key point in each deformation region through a key point generation module.

[0114] S1432: For each deformation area, Gaussian distribution technology is used in the key point generation module to calculate the influence of the three-dimensional coordinates of each deformation key point in the deformation area, obtain the characteristic influence value of each deformation key point in the three-dimensional image, and perform weighted summation on each characteristic influence value to form the deformation feature of the deformation area.

[0115] In this embodiment, after the standard face image is mapped to the three-dimensional latent space, the key point generation module can determine multiple deformation regions in the three-dimensional image and the three-dimensional coordinates of each deformation key point in each deformation region in the latent space. Then, for each deformation region, the computer device can use the Gaussian distribution technology to perform influence calculation on the three-dimensional coordinates of each deformation key point in the deformation region, obtain the feature influence value of each deformation key point in the three-dimensional image, and perform weighted summation on each feature influence value to form the deformation feature of the deformation region.

[0116] It is understandable that Gaussian distribution is also called normal distribution, which is used to describe the distribution of data around the mean value and has a symmetrical bell-shaped curve feature. In this application, the key point generation module can use Gaussian distribution to quantify the influence of the generated key points, ensuring that the deformation features calculated by each key point are based on the comprehensive influence of multiple points, rather than relying solely on a single key point, so as to ensure the smoothness and naturalness of the facial expression driving effect.

[0117] For example, the key point generation module can generate a set of key points with coordinates (x, y, z) in the latent space to control the position deformation of the deformation area features in the latent space. There can be 20 key points corresponding to each deformation area. Each key point has a direct impact on the image features around it in the three dimensions of x, y, and z. At this time, the key point generation module can use a Gaussian distribution to characterize the influence range and degree of each point. Finally, the influence of these points is weighted and summed to obtain the final deformation feature.

[0118] The character expression driving device provided in the embodiment of the present application is described below. The character expression driving device described below and the character expression driving method described above can be referenced to each other.

[0119] In one embodiment, Figure 5 As shown, Figure 5A structural diagram of a character expression driving device provided in an embodiment of the present application; the present application also provides a character expression driving device, including a data acquisition module 210, a face recognition module 220, a model determination module 230 and an expression driving module 240, specifically including the following:

[0120] The data acquisition module 210 is used to acquire a character video containing a human face and an expression driving target of the character video, and parse and obtain a video frame image of the character video.

[0121] The face recognition module 220 is used to use the target face detection model to identify the face key points in the video frame image, and perform face alignment on the video frame image based on the face key points to obtain a standard face image.

[0122] The model determination module 230 is used to determine the target expression driving model; the target expression driving model is composed of a three-dimensional convolutional neural network, which is used to map the two-dimensional image into a three-dimensional latent space for feature encoding and expression driving.

[0123] The expression driving module 240 is used to input the standard face image and the expression driving target into the target expression driving model, and use the target expression driving model to perform expression driving on the standard face image based on the expression driving target to output an expression driving video.

[0124] In the above embodiment, when performing expression driving on a person in a person video, the expression driving target of the person video can be determined first, and the continuous video frame images in the person video can be obtained by parsing, so that expression driving can be realized on the video frame images to ensure the naturalness of the expression driving effect; then the target face detection model can be used to identify the key points of the face in the video frame image, and the video frame image can be aligned based on the key points of the face to obtain a standard face image, so that the standard face image can be expression driven based on the expression driving target through the target expression driving model, and output as an expression driving video, so as to improve the quality of the expression driving effect. Since the target expression driving model is composed of a three-dimensional convolutional neural network, which is used to map a two-dimensional image into a three-dimensional latent space for feature encoding and expression driving, the target expression driving model can accurately track the face at various angles in the video frame image and perform expression editing and driving, so as to achieve the natural connection of the expression between frames of the video frame image without affecting the quality of the person video.

[0125] In one embodiment, the face recognition module 220 may include:

[0126] The model determines the sub-model, which is used to determine the target face detection model. The target face detection model includes a face recognition network and a key point detection network.

[0127] The face recognition sub-model is used to perform face recognition on the video frame image using the face recognition network to obtain the face position frame in the video frame image.

[0128] The key point detection sub-model is used to perform key point detection on the original face image based on the face position box in the key point detection network to obtain the face key points.

[0129] In one embodiment, the face recognition module 220 may further include:

[0130] The sample acquisition sub-model is used to obtain sample face images marked with real face position frames, and the real face position frames are marked with real face key points.

[0131] The model prediction sub-model is used to input the sample face image into the preset initial face detection model to obtain the predicted face position frame output by the initial face detection model for the sample face image, as well as the predicted confidence of the predicted face position frame and the predicted face key points.

[0132] The model training sub-model is used to train the initial face detection model with the goal of making the predicted face position frame and predicted face key points close to the real face position frame and real face key points respectively.

[0133] The model lightweight sub-model is used to lightweight the trained initial face detection model to obtain the target face detection model when the initial face detection model meets the preset training conditions.

[0134] In one embodiment, the model training sub-model includes:

[0135] The first loss value calculation unit is used to determine the position loss value of the predicted face position frame based on the real face position frame, and calculate the confidence loss value of the prediction confidence according to the position loss value.

[0136] The second loss value calculation unit is used to determine the key point loss value of the predicted face key point based on the real face key point.

[0137] The parameter updating unit is used to update the parameters in the initial face detection model according to the position loss value, the confidence loss value and the key point loss value.

[0138] In one embodiment, the face recognition module 220 may further include:

[0139] The position transformation sub-model is used to locate the face area in the video frame image according to the face key points, and use affine transformation technology to align the located face area to a preset standard position to form a standard face image.

[0140] In one embodiment, the target expression driving model in the expression driving module 240 includes a latent space mapping module, a key point generation module, a spatial local deformation module and a feature decoding module; the expression driving module 240 may include:

[0141] The latent space mapping sub-model is used to extract the low-order and high-frequency features of the standard face image using the latent space mapping module, and perform latent space encoding and feature decoupling on the low-order and high-frequency features to generate a three-dimensional image corresponding to the standard face image in the latent space.

[0142] The deformation feature determination sub-model is used to determine multiple deformation areas in the three-dimensional image and the deformation features corresponding to each deformation area through a key point generation module.

[0143] The feature editing sub-model is used to determine the deformation area and deformation direction corresponding to the expression-driven target using the spatial local deformation module, and to perform feature editing on the deformation features of the deformation area based on the deformation direction to generate an expression-driven image.

[0144] The feature decoding sub-model is used to input the expression-driven image into the feature decoding module, so that the feature decoding module uses a generator composed of a SPADE network to upsample and decode the expression-driven image to map the expression-driven image to a two-dimensional space and generate an expression-driven video.

[0145] In one embodiment, the feature editing sub-model may include:

[0146] The three-dimensional coordinate determination unit is used to determine the three-dimensional coordinates of multiple deformation areas in the three-dimensional image and each deformation key point in each deformation area through the key point generation module.

[0147] The deformation feature calculation unit is used to use Gaussian distribution technology in the key point generation module to perform influence calculation on the three-dimensional coordinates of each deformation key point in each deformation area, obtain the characteristic influence value of each deformation key point in the three-dimensional image, and perform weighted summation on each characteristic influence value to form the deformation feature of the deformation area.

[0148] In one embodiment, the present application also provides a storage medium, which stores computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the character expression driving method described in any of the above embodiments.

[0149] In one embodiment, the present application also provides a computer device, in which computer-readable instructions are stored. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the character expression driving method as described in any of the above embodiments.

[0150] Indicatively, Figure 6 As shown, Figure 6 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. The computer device 300 may be provided as a server. Figure 6 The computer device 300 includes a processing component 302, which further includes one or more processors, and a memory resource represented by a memory 301, for storing instructions executable by the processing component 302, such as an application. The application stored in the memory 301 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 302 is configured to execute instructions to perform the character expression driving method of any of the above embodiments.

[0151] The computer device 300 may further include a power supply component 303 configured to perform power management of the computer device 300, a wired or wireless network interface 304 configured to connect the computer device 300 to a network, and an input / output (I / O) interface 305. The computer device 300 may operate based on an operating system stored in the memory 301, such as Windows Server TM, Mac OS X TM, Unix TM, Linux TM, Free BSD TM, or the like.

[0152] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0153] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0154] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

[0155] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for driving a character's expression, characterized in that: The method comprises: Acquire a person video containing a human face and an expression driving target of the person video, and parse to obtain continuous video frame images in the person video; Using a target face detection model to identify key face points in the video frame image, and performing face alignment on the video frame image based on the key face points to obtain a standard face image; Determine a target expression driving model; the target expression driving model is composed of a three-dimensional convolutional neural network, which is used to map a two-dimensional image into a three-dimensional latent space for feature encoding and expression driving; The standard face image and the expression driving target are input into the target expression driving model, and the target expression driving model is used to perform expression driving on the standard face image based on the expression driving target to output an expression driving video.

2. The character expression driving method according to claim 1, characterized in that: The step of using the target face detection model to identify key face points in the video frame image includes: Determine a target face detection model, wherein the target face detection model includes a face recognition network and a key point detection network; Performing face recognition on the video frame image using the face recognition network to obtain a face position frame in the video frame image; In the key point detection network, key point detection is performed on the original face image based on the face position frame to obtain face key points.

3. The character expression driving method according to claim 1 or 2, characterized in that: The training process of the target face detection model includes: Acquire a sample face image marked with a real face position frame, wherein the real face position frame is marked with real face key points; Inputting the sample face image into a preset initial face detection model, obtaining a predicted face position frame output by the initial face detection model for the sample face image, and a predicted confidence level and predicted face key points of the predicted face position frame; Training the initial face detection model with the goal of making the predicted face position frame and the predicted face key points approach the real face position frame and the real face key points respectively; When the initial face detection model meets the preset training conditions, the trained initial face detection model is lightweighted to obtain a target face detection model.

4. The character expression driving method according to claim 3, characterized in that: The training of the initial face detection model with the goal of making the predicted face position frame and the predicted face key points approach the real face position frame and the real face key points respectively comprises: Determine a position loss value of the predicted face position frame based on the real face position frame, and calculate a confidence loss value of the prediction confidence according to the position loss value; Determine a key point loss value of the predicted face key point based on the real face key point; Update the parameters in the initial face detection model according to the position loss value, the confidence loss value and the key point loss value.

5. The character expression driving method according to claim 1, characterized in that: The step of performing face alignment on the video frame image based on the face key points to obtain a standard face image includes: The face region in the video frame image is located according to the face key points, and the located face region is aligned to a preset standard position using affine transformation technology to form a standard face image.

6. The character expression driving method according to claim 1, characterized in that: The target expression driving model includes a latent space mapping module, a key point generation module, a spatial local deformation module and a feature decoding module; The step of using the target expression driving model to perform expression driving on the standard face image based on the expression driving target includes: The latent space mapping module is used to extract low-order and high-frequency features of the standard face image, and the low-order and high-frequency features are latently encoded and feature decoupled to generate a three-dimensional image corresponding to the standard face image in the latent space; Determining a plurality of deformation regions in the three-dimensional image and a deformation feature corresponding to each deformation region by means of the key point generation module; Determine the deformation area and deformation direction corresponding to the expression-driven target by using the spatial local deformation module, and perform feature editing on the deformation features of the deformation area based on the deformation direction to generate an expression-driven image; The expression-driven image is input into the feature decoding module, so that the feature decoding module uses a generator composed of a SPADE network to upsample and decode the expression-driven image, so as to map the expression-driven image to a two-dimensional space and generate an expression-driven video.

7. The character expression driving method according to claim 6, characterized in that: Each deformation region is composed of multiple deformation key points; The determining of the plurality of deformation regions in the three-dimensional image and the deformation features corresponding to each deformation region by the key point generation module includes: Determine, by means of the key point generation module, a plurality of deformation regions in the three-dimensional image and the three-dimensional coordinates of each deformation key point in each deformation region; For each deformation area, Gaussian distribution technology is used in the key point generation module to calculate the influence of the three-dimensional coordinates of each deformation key point in the deformation area, obtain the characteristic influence value of each deformation key point in the three-dimensional image, and perform weighted summation on each characteristic influence value to form the deformation feature of the deformation area.

8. A character expression driving device, characterized in that: include: A data acquisition module, used to acquire a person video containing a human face and an expression driving target of the person video, and parse to obtain continuous video frame images in the person video; A face recognition module, used to identify key points of faces in the video frame image using a target face detection model, and perform face alignment on the video frame image based on the key points of faces to obtain a standard face image; A model determination module is used to determine a target expression driving model; the target expression driving model is composed of a three-dimensional convolutional neural network, which is used to map a two-dimensional image into a three-dimensional latent space for feature encoding and expression driving; The expression driving module is used to input the standard face image and the expression driving target into the target expression driving model, and use the target expression driving model to perform expression driving on the standard face image based on the expression driving target to output an expression driven video.

9. A storage medium, characterized in that: The storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the character expression driving method as described in any one of claims 1 to 7.

10. A computer device, characterized in that: include: one or more processors, and memory; The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the character expression driving method according to any one of claims 1 to 7 are performed.

Images