Driving digital human expression generation method

Abstract

The invention provides a driving digital human expression generation method, and relates to the technical field of digital human generation, and the generation method comprises the steps: obtaining a driving video and a driven video; the driving video is an expression providing object of the driven video; inputting the driving video into a target three-dimensional face reconstruction model to obtain a target face coefficient feature; inputting the target facial coefficient feature and the driven video into a target expression generation model to obtain a driven image with a target expression; performing video coding based on the driven image to obtain a target video; the target video is a video combining the face information of the driven video and the target expression of the driving video. Through the generation method, the problem that the driving digital human expression generated based on the prior art is too stiff is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of digital human generation, and in particular to a method for driving digital human expression generation. Background Art

[0002] The research on digital human expressions is an important part of the field of natural human-computer interaction. It aims to accurately apply facial expressions to digital humans, so that digital humans can present realistic emotional expressions, improve user experience, promote emotional resonance, and bring higher immersion and authenticity to human-computer interaction. Facial coefficient-driven digital human expressions are driven by obtaining facial expression coefficients in the driving video for encoding, and then driving the digital human to generate expressions.

[0003] At present, the research on driving digital human expressions through facial coefficients is mainly focused on the optimization of expression effects. Most facial coefficient-driven digital human technologies can achieve good migration effects on simple expressions such as smiles and happiness or face reconstruction, but cannot achieve migration of subtle, exaggerated, asymmetrical or continuous expression changes. In many application scenarios, unrealistic digital human expressions will lead to insufficient emotional communication, reduce the interactive experience, and make communication effects poor, which in turn affects the user's willingness to participate and overall satisfaction. At present, there is no method in the relevant technology that can realize the generation of digital human expressions driven by facial coefficients. This is the bottleneck that needs to be solved in driving digital human expressions with facial coefficients. Summary of the invention

[0004] The present application provides a method for generating driven digital human expressions, so as to solve the problem that the driven digital human expressions generated based on the prior art are too stiff.

[0005] The generation method comprises: Acquire a driving video and a driven video; the driving video is an expression providing object of the driven video; Inputting the driving video into a target three-dimensional face reconstruction model to obtain target facial coefficient features; the target three-dimensional face reconstruction model extracts features from the driving video based on a skip connection method; Inputting the target facial coefficient feature and the driven video into a target expression generation model, wherein the target expression generation model is configured to perform feature global analysis and feature local analysis according to the target facial coefficient feature and the driven video to obtain a driven image with a target expression; Video encoding is performed based on the driven image to obtain a target video; the target video is a video that combines the facial information of the driven video and the target expression of the driving video.

[0006] Preferably, the training process of the target 3D face reconstruction model includes: Acquire an expression video, and perform a first preprocessing on the expression video to obtain a target expression video; Model training is performed according to the target expression video and the three-dimensional face reconstruction model and convergence is performed to obtain the target three-dimensional face reconstruction model.

[0007] Preferably, the step of performing model training according to the target expression video and the three-dimensional face reconstruction model comprises: Selecting a plurality of frames of training expression images from the target expression video; Performing a first processing on the training expression image to obtain a facial coefficient feature; The training of the three-dimensional face reconstruction model is repeated, and the facial coefficient features are converged using a first loss function until the three-dimensional face reconstruction model meets the first requirement, thereby obtaining the target three-dimensional face reconstruction model.

[0008] Preferably, the step of performing a first processing according to the training expression image comprises: Reconstructing and masking the training expression image respectively to obtain a first reconstructed feature and a first masked image; Performing a jump-type feature fusion process according to the first reconstructed feature and the first cover image to obtain a first fused feature; The first fusion feature is decoded to obtain the facial coefficient feature.

[0009] Preferably, the step of reconstructing the training expression image comprises: Performing encoding processing on the training expression image to obtain a first encoding feature; Perform three-dimensional face mesh reconstruction on the first coding feature to obtain a first reconstructed feature.

[0010] Preferably, the step of using the first loss function to converge the facial coefficient features comprises: Calculating a first error between the facial coefficient feature and the training expression image using an L1 loss function; Calculating a second error between the facial coefficient feature and the training expression image using a cycle loss function; When both the first error and the second error meet a first requirement, the training of the three-dimensional face reconstruction model is stopped to obtain the target three-dimensional face reconstruction model.

[0011] Preferably, the target 3D face reconstruction model includes a first convolutional coding unit, an image masking unit, a 3D face mesh reconstruction unit, a feature fusion unit and a second convolutional coding unit; The feature fusion unit includes a coding layer and a decoding layer, the coding layer includes a first coding layer, a second coding layer and a third coding layer connected in sequence, the decoding layer includes a first decoding layer, a second decoding layer and a third decoding layer connected in sequence, the third coding layer is connected to the first decoding layer; the coding layer and the decoding layer are jump-connected; The first encoding layer, the second encoding layer and the third encoding layer are all configured to downsample the features of the previous layer input; the first decoding layer, the second decoding layer and the third decoding layer are all configured to upsample the features of the previous layer input.

[0012] Preferably, the training process of the target expression generation model includes: Acquire a training drive video, and perform a second preprocessing on the training drive video to obtain a target training drive video; Model training is performed and converged according to the target training driving video, the facial coefficient features and the expression generation model to obtain the target expression generation model.

[0013] Preferably, the step of performing model training according to the target training driving video, the facial coefficient features and the expression generation model comprises: Performing convolution coding on the target training driving video and the facial coefficient feature respectively to obtain a second coding feature and a third coding feature; The second coding feature and the third coding feature are subjected to feature fusion by a second process to obtain a second fused feature; the second process comprises an adaptive normalization process with an attention mechanism; Performing residual calculation on the second fused feature to obtain a first residual feature; Performing deconvolution processing on the first residual feature to obtain a first deconvolution feature; Performing a third processing on the first deconvolution feature to obtain a training driven image; the third processing includes extracting deep features from the first deconvolution feature by using a spatial normalization process, and uniformly and adaptively adjusting a normalization parameter of the spatial normalization process; Performing video encoding according to the training driven image to obtain a training target video; The training target video is converged by using an L1 loss function until the expression generation model meets the second requirement, thereby obtaining the target expression generation model.

[0014] Preferably, the second process comprises: The second coding feature and the third coding feature are divided into a plurality of subsequences, feature coding processing is performed on each subsequence, attention weights corresponding to different subsequences are calculated by a spatial attention module, and a plurality of feature coding processing results are fused based on the attention weight calculation results to obtain local feature coding; the spatial attention module is preset in the expression generation model; The local feature code is divided into multiple subsequences, feature coding is performed on each subsequence, attention weights corresponding to different subsequences are calculated by the spatial attention module, and multiple feature coding processing results are fused based on the attention weight calculation results to obtain the local feature code; Iteratively updating, and independently outputting the local feature codes obtained in each round to obtain multiple local feature codes; Attention weights are calculated for the multiple local feature codes according to the channel attention module, and feature processing is performed based on the attention weight calculation results to obtain a global feature code; the multiple local feature codes are fused with the global feature code to obtain the second fused feature; the channel attention module is preset in the expression generation model.

[0015] Preferably, the step of using the L1 loss function to converge the training generated video includes: Calculating a third error between the training target video and the target expression video using the L1 loss function; When the third error meets the second requirement, the training of the expression generation model is stopped to obtain the target expression generation model.

[0016] As can be seen from the above content, the present application provides a method for generating expression of a driven digital human, the generating method comprising obtaining a driving video and a driven video; the driving video is the expression providing object of the driven video; the driving video is input into a target three-dimensional face reconstruction model to obtain a target facial coefficient feature; the target facial coefficient feature and the driven video are input into a target expression generating model to obtain a driven image with a target expression; video encoding is performed based on the driven image to obtain a target video; the target video is a video combining the facial information of the driven video and the target expression of the driving video. The present application solves the problem that the expression of a driven digital human generated based on the prior art is too stiff through the above generating method. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1 A flowchart of a method for driving digital human expression generation in this application; Figure 2 A training flow chart of a three-dimensional face reconstruction model in a method for generating digital human expressions driven by this application; Figure 3 A specific flow chart for training a three-dimensional face reconstruction model in a method for driving digital human expression generation in this application; Figure 4 This is a flowchart of obtaining facial coefficient features in a method for driving digital human expression generation in this application; Figure 5 This is a flow chart of the convergence method of a three-dimensional face reconstruction model in a method for driving digital human expression generation in this application; Figure 6 A schematic diagram of a target three-dimensional face reconstruction model in a method for driving digital human expression generation in this application; Figure 7 Schematic diagram of the feature fusion unit in the target 3D face reconstruction model; Figure 8 A training flow chart of an expression generation model in a method for driving digital human expression generation in this application; Fig. 9 A specific flow chart for training an expression generation model in a method for driving digital human expression generation in this application; Fig.10 A flow chart of the convergence mode of the expression generation model in the method for driving the expression generation of digital human in this application; Fig.11 This is a face comparison diagram before and after masking processing in a method for driving digital human expression generation in this application; Fig.12 A schematic diagram of an expression generation model in a method for driving digital human expression generation in the present application; Fig.13 This is a flowchart of the second process in a method for driving digital human expression generation in the present application. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] Figure 1 This is a flow chart of a method for driving digital human expression generation in this application.

[0021] See also Figure 1 It can be seen that this embodiment provides a method for driving digital human expression generation, and the generation method includes: S10, obtaining a driving video and a driven video. Specifically, in this embodiment, before generating a digital human, it is necessary to obtain a driving object and a driven object of the digital human, wherein the driving video is an expression providing object of the driven video.

[0022] Among them, the driving video is a recorded video of an actual user, and the driven video can be a virtual anchor image video.

[0023] It should be noted that the driven video can only be obtained after obtaining permission. It can be understood that some virtual character image videos have a usage license, so they must be used after obtaining permission.

[0024] The generating method further comprises: S20, inputting the driving video into the target three-dimensional face reconstruction model. Specifically, in this embodiment, before generating the expression video, it is necessary to first reconstruct the driving video to improve the accuracy and detail richness of the facial expression image of the driving video.

[0025] In this embodiment, the driving video is input into the target three-dimensional face reconstruction model to perform face reconstruction processing, so as to obtain the features of the reconstructed face, that is, the target facial coefficient features.

[0026] The generating method further comprises: S30, input the target facial coefficient features and the driven video into the target expression generation model. Specifically, in the present embodiment, in the conventional driving digital human expression generation technology, the expression provider and the expression receiver are generally directly transferred, which makes the generated expressions often appear incoherent, uncoordinated and other problems; in the above step S20, the face of the expression provider, that is, the driving video, has been reconstructed, and if the generated expression is to be made more natural, it is necessary to perform certain processing on the expression receiver, and at the same time, perform corresponding facial processing on the expression provider and the expression receiver, so as to further improve the authenticity of the generated expression.

[0027] Among them, in this embodiment, since it is necessary to migrate the expression from one video object to another video object, the video object cannot be processed accordingly alone; for this, this embodiment obtains the target facial coefficient feature obtained in step S20 and the driven video for receiving the expression into the target expression generation model to migrate the expression, thereby obtaining an image after expression migration, that is, a driven image with the target expression.

[0028] The generating method further comprises: S40, performing video encoding based on the driven image. Specifically, in the present embodiment, in step S30, a driven image with a migrated expression has been obtained, and a single image cannot constitute a complete video. Therefore, it is necessary to perform video encoding on the driven image to convert the image into a video to obtain the target video.

[0029] The target video is a video combining the facial information of the driven video and the target expression of the driving video.

[0030] Figure 2 This application discloses a training flowchart of a three-dimensional face reconstruction model in a method for driving digital human expression generation.

[0031] See also Figure 2 It can be seen that, further, in some embodiments, the training process of the target 3D face reconstruction model includes: S100, obtaining an expression video, and performing a first preprocessing on the expression video. Specifically, in the present embodiment, since the target three-dimensional face reconstruction model is used to perform expression reconstruction and feature extraction on the expression provider, the training of the target three-dimensional face reconstruction model also requires obtaining the corresponding expression video. The videos include various types, but the target three-dimensional face reconstruction model needs to meet the requirements of complete and clear expressions.

[0032] Among them, since the expression video obtained for training is generally obtained from an open source video database, and the quality of such video cannot perfectly meet the requirements of the training data required for the target three-dimensional face reconstruction model, it is necessary to perform the first preprocessing on the expression video to obtain a video that meets the training requirements of the target three-dimensional face reconstruction model.

[0033] Wherein, the first preprocessing may include data enhancement operations such as face detection, key point detection, random brightness, contrast, color jitter, Gaussian noise and blur processing according to different situations. It should be noted that in the training process of the target three-dimensional face reconstruction model, the expression-providing video object needs to be preprocessed, while in the actual application of the target three-dimensional face reconstruction model, the expression-providing video object does not need to be preprocessed. The data is preprocessed in the training stage in order to better train the model, and the steps in the first preprocessing are not necessary conditions for model training.

[0034] After performing the first preprocessing on the expression video, a preprocessed target expression video is obtained.

[0035] S200, performing model training and convergence according to the target expression video and the three-dimensional face reconstruction model. Specifically, in the present embodiment, in the process of performing model training on the three-dimensional face reconstruction model using the expression video, it is necessary to perform loss convergence on the three-dimensional face reconstruction model to obtain the target three-dimensional face reconstruction model that meets the target requirements.

[0036] Figure 3 This is a specific flow chart for training a three-dimensional face reconstruction model in a method for driving digital human expression generation in this application.

[0037] See also Figure 3 It can be seen that, further, in some embodiments, the step of performing model training according to the target expression video and the three-dimensional face reconstruction model includes: S210, selecting several frames of training expression images from the target expression video. Specifically, in this embodiment, during the training process of the three-dimensional face reconstruction model, it is necessary to select certain image frames from the target expression video and perform training based on the image frames.

[0038] The step of performing model training according to the target expression video and the three-dimensional face reconstruction model also includes: S220, performing a first processing according to the training expression image. Specifically, in this embodiment, the target expression video is input into the three-dimensional face reconstruction model, and the three-dimensional face reconstruction model is used to perform a first processing on several image frames of the target expression video to obtain facial coefficient features corresponding to the face in the target expression video.

[0039] The step of performing model training according to the target expression video and the three-dimensional face reconstruction model also includes: S230, repeat the training of the 3D face reconstruction model, and use the first loss function to converge the facial coefficient features until the 3D face reconstruction model meets the first requirement. Specifically, in this embodiment, the facial coefficient features obtained in the above step S220 are the output of the model, but if the model is to be trained, it is necessary to continuously perform loss convergence on the facial coefficient features to continuously improve the 3D face reconstruction model.

[0040] Among them, during the training process of the three-dimensional face reconstruction model, the first loss function is used to calculate the loss of the facial coefficient features output each time, so as to achieve the convergence of the three-dimensional face reconstruction model and finally obtain the target three-dimensional face reconstruction model.

[0041] Figure 4 This application discloses a flowchart for obtaining facial coefficient features in a method for generating digital human expressions.

[0042] See also Figure 4 It can be seen that, further, in some embodiments, the step of performing a first processing according to the training expression image includes: S221, respectively reconstructing and masking the training expression images. Specifically, in the present embodiment, the first step of the first processing includes respectively reconstructing and masking the training expression images, which can be understood as dividing the training expression images into two sequences, reconstructing one sequence, and masking the other sequence; wherein the training expression images subjected to reconstruction obtain a first reconstructed feature; and the training expression images subjected to masking obtain a first masked image.

[0043] The reconstruction process includes encoding the training expression image to obtain a first encoding feature; and reconstructing a three-dimensional face mesh on the first encoding feature to obtain a first reconstructed feature.

[0044] The image after the masking process and the image before the masking process can be seen in Fig.11 , see Fig.11 It can be seen that the masking process is to mask the more important areas of the face of the face image. The reason is that the acquisition of facial features needs to satisfy the characteristics of the entire face. Therefore, it is necessary to blacken the important feature areas of the entire face, so as to provide an effective data basis for subsequent feature fusion.

[0045] The step of performing a first process according to the training expression image also includes: S222, performing a jump-type feature fusion processing according to the first reconstructed feature and the first cover image. Specifically, in this embodiment, the first reconstructed feature and the first cover image obtained in step S221 are subjected to feature fusion processing, wherein the feature fusion processing is a jump-type fusion processing, and its principle is based on a neural network structure of shortcut connection.

[0046] The first fusion feature is obtained by performing a jump-type feature fusion process on the first reconstructed feature and the first cover image.

[0047] The step of performing a first process according to the training expression image also includes: S223, encoding the first fusion feature. Specifically, in this embodiment, after completing step S222, the first fusion feature obtained in step S222 is encoded to obtain a corresponding feature, that is, the facial coefficient feature.

[0048] It should be noted that although the above steps S221 to S223 are the process of model training, they are also the same as steps S221 to S223 in the actual application of the model.

[0049] Figure 5 This is a flow chart of the convergence method of a three-dimensional face reconstruction model in a method for driving digital human expression generation in this application.

[0050] See also Figure 5 It can be seen that, further, in some embodiments, the step of using the first loss function to converge the facial coefficient feature includes: S231, calculating a first error between the facial coefficient feature and the training expression image using an L1 loss function; S232, calculating a second error between the facial coefficient feature and the training expression image using a cycle loss function; Specifically, in this embodiment, the error between the facial coefficient feature and the training expression image is calculated by simultaneously using the L1 loss function and the cycle loss function, so as to obtain the first error and the second error.

[0051] The step of using the first loss function to converge the facial coefficient features also includes: S233, when the first error and the second error both meet the first requirement, stop training the three-dimensional face reconstruction model. Specifically, in this embodiment, when the first error and the second error both meet the first requirement, it means that the three-dimensional face reconstruction model has completed training, so the training can be stopped and the target three-dimensional face reconstruction model is obtained.

[0052] Among them, we use the L1 loss function to calculate the L1 reconstruction error between the facial coefficient feature and the training expression image. In order to make the predicted facial coefficient more accurate and stable, we add a cycle loss function, that is, the cycle loss between the facial coefficient feature and the facial coefficient of the training expression image.

[0053] Figure 8 This application discloses a training flow chart of an expression generation model in a method for driving digital human expression generation.

[0054] See also Figure 8 It can be seen that, further, in some embodiments, the training process of the target expression generation model includes: S300, obtaining a training drive video, and performing a second preprocessing on the training drive video. Specifically, in the present embodiment, since the target expression generation model is used to extract expression features of the expression recipient and the expression provider, the training of the target expression generation model also requires obtaining the corresponding training drive video. Since the training drive video is the same as the expression video and is diverse, the target expression generation model also needs to meet certain requirements.

[0055] Among them, since the training drive video obtained for training can be obtained from an open source video database or recorded by the training model personnel, and the quality of these videos is also uneven, it is necessary to perform a second preprocessing on the training drive video to obtain a video that meets the training requirements of the target expression generation model.

[0056] Among them, the second preprocessing is quite different from the first preprocessing, and the second preprocessing may include video beautification, face detection and cropping according to different situations. It should be noted that in the training process of the target expression generation model, the expression receiving video object needs to be preprocessed, but in the actual application of the target expression generation model, the expression receiving party does not need to be preprocessed. The data is preprocessed in the training stage in order to better train the model, and the steps in the second preprocessing are not necessary conditions for model training.

[0057] After the training drive video is subjected to a second preprocessing, a preprocessed target training drive video is obtained.

[0058] The training process of the target expression generation model also includes: S400, model training is performed and converged according to the target training driving video, the facial coefficient features and the expression generation model. Specifically, in the present embodiment, in the process of training the expression generation model using the target training driving video and the facial coefficient features, it is also necessary to perform loss convergence on the expression generation model, so as to obtain a loss target expression generation model that meets the target requirements.

[0059] Fig. 9 This is a specific flow chart of the training of an expression generation model in a method for driving digital human expression generation in this application.

[0060] See also Fig. 9 It can be seen that, further, in some embodiments, the step of performing model training and convergence according to the target training driving video, the facial coefficient features and the expression generation model includes: S410, convolution encoding is performed on the target training drive video and the facial coefficient feature respectively to obtain a second encoding feature and a third encoding feature. Specifically, in this embodiment, convolution encoding is performed on the target training drive video and the facial coefficient feature respectively to obtain the second encoding feature and the third encoding feature, wherein the second encoding feature corresponds to the target training drive video, and the third encoding feature corresponds to the facial coefficient feature.

[0061] The step of performing model training and convergence according to the target training driving video, the facial coefficient features and the expression generation model also includes: S420, performing feature fusion on the second coding feature and the third coding feature using a second process to obtain a second fused feature; the second process includes an adaptive normalization process with an attention mechanism; S430, performing residual calculation on the second fused feature to obtain a first residual feature; S440, performing deconvolution processing on the first residual feature to obtain a first deconvolution feature; S450, performing a third processing on the first deconvolution feature to obtain a training driven image; the third processing includes extracting deep features from the first deconvolution feature by using a spatial normalization process, and uniformly and adaptively adjusting a normalization parameter of the spatial normalization process; Specifically, in this embodiment, through steps S420 to S450, an improved neural network is used to perform correlation processing on the coding features of the expression provider and the coding features of the expression receiver to merge the two together and enhance the correlation between the face and the expression, thereby enhancing the authenticity of the subsequently generated expression video.

[0062] The step of performing model training and convergence according to the target training driving video, the facial coefficient features and the expression generation model also includes: S460, performing video encoding according to the training driven image to obtain a training target video; S470, using an L1 loss function to converge the training target video until the expression generation model meets the second requirement, thereby obtaining the target expression generation model.

[0063] Specifically, in this embodiment, after completing the processing from step S420 to step S450, the features are converted into videos, and the generated videos are continuously subjected to loss convergence, so as to finally obtain the target expression generation model.

[0064] Fig.10 This is a flow chart of the convergence method of the expression generation model in the method for driving digital human expression generation in the present application.

[0065] See also Fig.10 It can be seen that, further, in some embodiments, the step of using the L1 loss function to converge the training generated video includes: S471, calculating a third error between the training target video and the target expression video using the L1 loss function; S472, when the third error meets the second requirement, stop training the expression generation model to obtain the target expression generation model.

[0066] Specifically, in this embodiment, similar to the convergence method of the three-dimensional face reconstruction model, the loss convergence of the expression generation model uses the L1 loss function to calculate the third error between the training target video and the target expression video, and judges whether the expression generation model is trained based on the error.

[0067] Figure 6 A schematic diagram of a target three-dimensional face reconstruction model in a method for driving digital human expression generation in this application; Figure 7 Schematic diagram of the feature fusion unit in the target 3D face reconstruction model.

[0068] See also Figure 6 and Figure 7 It can be seen that, further, in some embodiments, the target three-dimensional face reconstruction model includes a first convolutional coding unit, an image masking unit, a three-dimensional face mesh reconstruction unit, a feature fusion unit and a second convolutional coding unit; The feature fusion unit includes a coding layer and a decoding layer, the coding layer includes a first coding layer, a second coding layer and a third coding layer connected in sequence, the decoding layer includes a first decoding layer, a second decoding layer and a third decoding layer connected in sequence, the third coding layer is connected to the first decoding layer; the coding layer and the decoding layer are jump-connected; The first encoding layer, the second encoding layer and the third encoding layer are all configured to downsample the features of the previous layer input; the first decoding layer, the second decoding layer and the third decoding layer are all configured to upsample the features of the previous layer input.

[0069] Specifically, in this embodiment, the target three-dimensional face reconstruction model takes the target facial coefficient features and the driven video as data input, obtains the facial coefficients through the first convolution coding unit, and reconstructs the three-dimensional face mesh through the three-dimensional face mesh reconstruction unit; extracts the target facial coefficient features through the feature fusion unit and the second convolution coding unit.

[0070] The feature fusion unit is composed of an improved neural network structure, including 3 encoding layers for downsampling and 3 decoding layers for upsampling. Each corresponding layer is spliced ​​using shortcuts (jump connections). The feature fusion unit takes the output of the image masking unit and the output of the 3D face mesh reconstruction unit as input, and can better capture global information such as the edge information, face texture, and expression of the face through residual and jump connections. Specifically, the above 3 encoding layers gradually reduce the spatial dimension of the image while increasing the number of feature channels; correspondingly, the 3 decoding layers gradually restore the spatial dimension of the image while reducing the number of feature channels; the 3 encoding layers correspond to the 3 decoding layers. In addition to being transmitted to the next encoding layer, the output of each encoding layer is also directly transmitted to the corresponding decoding layer (the data transmission relationship here can be understood as the above jump connection), thereby ensuring information fusion and alleviating the gradient disappearance problem.

[0071] Fig.12 This is a schematic diagram of an expression generation model in a method for driving digital human expression generation in the present application.

[0072] Fig.13 This is a flowchart of the second process in a method for driving digital human expression generation in the present application.

[0073] See also Fig.12 as well as Fig.13 It can be seen that, further, in some embodiments, the second processing includes: S421, divide the second coding feature and the third coding feature into multiple subsequences, perform feature coding processing on each subsequence, calculate the attention weights corresponding to different subsequences through the spatial attention module, and fuse the multiple feature coding processing results based on the attention weight calculation results to obtain local feature coding; the spatial attention module is preset in the expression generation model. Specifically, in this embodiment, the spatial attention mechanism is introduced in the calculation process of local features by sequentially performing sequence division, attention weight calculation and local feature coding according to the attention weight on the second coding feature and the third coding feature. The spatial attention mechanism focuses on the important points of each spatial position in the feature map, so that better presentation results are achieved when calculating the attention of the local area.

[0074] Wherein, dividing the second coding feature and the third coding feature into a plurality of subsequences further includes the following operations: Sequence division is performed on the second coding feature and the third coding feature respectively, and finally the results of the two are normalized; or, the second coding feature and the third coding feature are normalized in the early stage, and then the fused features are sequenced.

[0075] It should be noted that the division operations of the second coding feature and the third coding feature may be different according to different application scenarios, but the preferred solution is to perform sequence division on the second coding feature and the third coding feature respectively, and finally normalize the results of the two.

[0076] The second process further includes: S422, divide the local feature code into multiple subsequences, perform feature coding processing on each subsequence, calculate the attention weights corresponding to different subsequences through the spatial attention module, and fuse multiple feature coding processing results based on the attention weight calculation results to obtain local feature code. Specifically, in this embodiment, the same as the above-mentioned step S421, both perform local feature analysis on the feature code, but the difference from step S421 is that after completing step S422, the obtained feature code needs to be iteratively updated, and the updated feature code is subjected to cyclic local feature analysis, and the obtained local feature code is independently output after each round of local feature analysis.

[0077] The second process further includes: S423, calculating the attention weights of the multiple local feature codes according to the channel attention module, and performing feature processing based on the attention weight calculation result to obtain the global feature code; fusing the multiple local feature codes with the global feature code to obtain the second fused feature; the channel attention module is preset in the expression generation model. Specifically, in this embodiment, after completing several rounds of local feature analysis, several local feature codes can be obtained. In step S423, it is necessary to perform further feature analysis on these local feature codes. The core purpose of step S423 is to introduce the channel attention mechanism in the feature analysis process. The channel attention mechanism focuses on calculating the importance of different feature channels relative to the global, so it has a better presentation in calculating the effect of global features.

[0078] Through steps S421 to S423, local analysis and global analysis of features are implemented, thereby enhancing the understanding of features by the expression generation model.

[0079] The above target 3D face reconstruction model helps to improve the accuracy and detail richness of the reconstructed facial expression images; at the same time, the feature fusion unit also has other advantages: Faster convergence speed: The introduction of residual structure makes it easier to converge; Stronger generalization ability: Multi-scale feature fusion integrates local and global information, which helps to enhance the generalization ability of the model and generate more natural and reasonable images.

[0080] The training process uses pre-training technology and trains 20 rounds on 1 million big data of open source expression video data and rich expression video data collected from the Internet, ensuring the accuracy of facial coefficient extraction and the generalization of the model. The three-dimensional face reconstruction model is improved through the feature fusion unit and the second convolutional coding unit. Multi-task regression training is performed on the face and the facial coefficients of the face on the collected diverse expression video big data, so that the model can accurately extract the facial coefficients of the expressive face.

[0081] Correspondingly, the specific structure, training and functions of the target expression generation model are as follows: During the training process of the target expression generation model, the driving person video is first input based on the aforementioned target three-dimensional face reconstruction model to obtain the corresponding facial coefficients, and then the obtained facial coefficients and the driven person image are used as input to output the corresponding expression image of the driven person, and the expression images are combined to form the final driven person expression video; the facial coefficients are input into the convolution module for encoding to obtain expression features; similarly, the driven person image is input into the convolution module for encoding to obtain facial features; the above expression features and facial features are input into the adaptive attention normalization module for feature fusion.

[0082] The adaptive attention normalization module is an upgraded version of the attention normalization module. Conventional adaptive normalization does not take local feature statistics into consideration and easily outputs unnatural facial expressions and local distortions. The adaptive attention normalization module introduces the attention mechanism to dynamically adjust the correlation between expression features and facial features, and performs adaptive weighting and feature fusion to reduce information loss.

[0083] Specifically, in the process of convolution encoding of facial coefficients and the driven person image, the shallow features (i.e., the output of the convolution layer at the front position in the hierarchical relationship) and the deep features (i.e., the output of the convolution layer at the back position in the hierarchical relationship) are simultaneously input into the adaptive attention normalization module as corresponding expression features and face features, so that the adaptive attention normalization module can comprehensively consider the deep and shallow features of the corresponding image, and then learn the spatial information of expressions and faces, especially better capture local details.

[0084] On this basis, the adaptive attention normalization module first calculates the attention weight distribution map based on the above-mentioned expression features and facial features containing spatial information. The weight distribution map is used to characterize which areas of the facial features are most critical for achieving expression changes (such as eyes, mouth, etc., these parts are usually most sensitive to expression changes, and different expressions will further lead to different attention weights between some details in the above-mentioned parts).

[0085] On this basis, the similarity between expression features and facial features can be further measured (cosine similarity and other methods), and based on the similarity between expression features and facial features, the aforementioned attention weights can be further adjusted.

[0086] After completing the above adjustments, the calculated attention weights are used to weight the facial features to enhance the features related to the expression features. On this basis, the weighted facial features are merged with the expression features to generate a new feature representation that contains both the identity information of the facial features and the changes in the target expression.

[0087] The advantages of the above adaptive attention normalization module also include: Improve model capabilities: Due to the introduction of the attention mechanism, the adaptive attention normalization module can more finely control feature fusion and generation results, thereby improving the quality and details of the generated images and making the output more coherent and natural.

[0088] Improved robustness: The adaptive attention normalization module adaptively adjusts the weights of feature fusion through the attention mechanism, better focuses on important features, is more robust to input perturbations, and enhances the generalization and robustness of the model.

[0089] The above fusion features are input into the residual network module and the deconvolution module. The residual network module learns the residual through skip residual connection, which can effectively avoid gradient vanishing, capture deep features, accelerate model convergence speed and improve the generalization ability of the model; while the deconvolution can ensure that more effective information is retained during the upsampling process, which can better restore high-resolution image details and improve the image generation quality; The output features of the above deconvolution are input into the spatial normalization module. The spatial normalization module is mainly composed of 5 normalized residual layers. The parameter adjustment of each normalized residual layer is learned using the spatial normalization module; the normalized residual layer is similar to batch normalization, and normalization is performed in a channel manner when activated; this module can mainly further extract deep features, and at the same time, the normalization parameters are adaptively adjusted through the spatial normalization module, and different features are applied to different positions of the expression face, which can better retain image details, avoid blur and distortion problems, make the generated expression face more accurate and natural, and can better reconstruct facial texture details.

[0090] Through the above-mentioned self-developed expression generation model, the image reconstruction of the driven person's face and the corresponding driving person's expression can be realized, and accurate facial expression migration and facial texture reconstruction can be achieved.

[0091] For the training of the above expression generation model, we use the L1 loss function. We calculate the L1 error between the real expression picture and the predicted expression picture to make the predicted expression more accurate. At the same time, in order to ensure that the expression changes of the generated face are more natural and the facial texture is more accurate, we add dynamic facial area mask enhancement and corresponding area L1 error calculation, and set its weight to 10. The above corresponds to two face discriminators, which perform L1 reconstruction loss and discrimination loss on the expression picture respectively, making the face, corresponding expression and facial texture more accurate and natural.

[0092] The above training also uses pre-training technology, extracting facial coefficients based on 3 million expression video big data and pre-training for 20 rounds. For each user, a 5-minute video with diverse expressions can be provided for fine-tuning. Only 5 rounds of training are required, and the rest remain the same, which can reduce training costs and achieve personalized customization effects.

[0093] When the above method is actually applied to generate a driving digital human, the driven person video is decoded into an image frame sequence, and the driving person video stream is obtained through a video acquisition device. During training, the driving person and the driven person are the same person mainly to regress the expression image and facilitate loss calculation. During testing, the driving person and the driven person can be different people or the same person. An improved three-dimensional face reconstruction model is used to extract the facial coefficients of the driving video, and the driven person image is input into the trained expression generation model to output a driven person image with the corresponding expression. Continuous expression images are encoded into videos for display by the terminal device.

[0094] Exemplarily, in this exemplary embodiment, a virtual psychotherapist is used as an example to upload a beautified video of the driven person and a video of the driving person to train the expression generation model in the cloud. After the training is completed, the facial coefficients of the driving person are extracted in real time, and combined with the video frames of the driven person to input into the model, so as to realize the generation of the corresponding expression of the driven image. At the same time, with the audio-driven lip-sync related technology, the effect can be displayed through streaming, which can provide patients with rich emotional feedback and enhance interaction and communication.

[0095] It should be noted that the solution provided in this embodiment can be applied as an independent expression migration function to live streaming, virtual chatting, virtual diagnosis and treatment, or short video creation, and can also be integrated into hardware devices as an extended function.

[0096] This embodiment has the following advantages: By improving the three-dimensional face reconstruction model, the facial coefficients of faces with diverse expressions can be extracted more accurately.

[0097] By introducing the attention mechanism, the correlation between expression features and facial features is dynamically adjusted, adaptive weighting and feature fusion are performed, and important features are focused on to avoid local distortion.

[0098] By extracting deep features and combining them with spatially adaptive normalization, different features are applied to different positions of the expressive face, which can better preserve image details and make the generated expressive faces and facial texture details more accurate and natural.

Claims

1. A method for driving digital human expression generation, characterized in that: The generation method comprises: Acquire a driving video and a driven video; the driving video is an expression providing object of the driven video; Inputting the driving video into a target three-dimensional face reconstruction model to obtain target facial coefficient features; the target three-dimensional face reconstruction model extracts features from the driving video based on a skip connection method; Inputting the target facial coefficient feature and the driven video into a target expression generation model, wherein the target expression generation model is configured to perform feature global analysis and feature local analysis according to the target facial coefficient feature and the driven video to obtain a driven image with a target expression; Video encoding is performed based on the driven image to obtain a target video; the target video is a video that combines the facial information of the driven video and the target expression of the driving video.

2. A method for generating expression of a digital human according to claim 1, characterized in that: The training process of the target 3D face reconstruction model includes: Acquire an expression video, and perform a first preprocessing on the expression video to obtain a target expression video; Model training is performed according to the target expression video and the three-dimensional face reconstruction model and convergence is performed to obtain the target three-dimensional face reconstruction model.

3. A method for generating expression of a digital human according to claim 2, characterized in that: The step of performing model training according to the target expression video and the three-dimensional face reconstruction model comprises: Selecting a plurality of frames of training expression images from the target expression video; Performing a first processing on the training expression image to obtain a facial coefficient feature; The training of the three-dimensional face reconstruction model is repeated, and the facial coefficient features are converged using a first loss function until the three-dimensional face reconstruction model meets the first requirement, thereby obtaining the target three-dimensional face reconstruction model.

4. A method for generating expression of a digital human according to claim 3, characterized in that: The step of performing a first processing according to the training expression image comprises: Reconstructing and masking the training expression image respectively to obtain a first reconstructed feature and a first masked image; Performing a jump-type feature fusion process according to the first reconstructed feature and the first cover image to obtain a first fused feature; The first fusion feature is encoded to obtain the facial coefficient feature.

5. A method for generating expression of a digital human according to claim 4, characterized in that: The step of reconstructing the training expression image comprises: Performing encoding processing on the training expression image to obtain a first encoding feature; Perform three-dimensional face mesh reconstruction on the first coding feature to obtain a first reconstructed feature.

6. A method for generating expression of a digital human according to claim 4, characterized in that: The step of using the first loss function to converge the facial coefficient features comprises: Calculating a first error between the facial coefficient feature and the training expression image using an L1 loss function; Calculating a second error between the facial coefficient feature and the training expression image using a cycle loss function; When both the first error and the second error meet a first requirement, the training of the three-dimensional face reconstruction model is stopped to obtain the target three-dimensional face reconstruction model.

7. A method for generating expression of a digital human according to claim 1, characterized in that: The target 3D face reconstruction model includes a convolutional coding unit, an image covering unit, a 3D face mesh reconstruction unit, a feature fusion unit and a second convolutional coding unit; The feature fusion unit includes a coding layer and a decoding layer, the coding layer includes a first coding layer, a second coding layer and a third coding layer connected in sequence, the decoding layer includes a first decoding layer, a second decoding layer and a third decoding layer connected in sequence, and the third coding layer is connected to the first decoding layer; There is a jump connection between the encoding layer and the decoding layer; The first encoding layer, the second encoding layer and the third encoding layer are all configured to downsample the features of the previous layer input; the first decoding layer, the second decoding layer and the third decoding layer are all configured to upsample the features of the previous layer input.

8. The method for generating expression of a digital human according to claim 3, characterized in that: The training process of the target expression generation model includes: Acquire a training drive video, and perform a second preprocessing on the training drive video to obtain a target training drive video; Model training is performed and converged according to the target training driving video, the facial coefficient features and the expression generation model to obtain the target expression generation model.

9. A method for generating expression of a digital human according to claim 8, characterized in that: The step of performing model training and convergence according to the target training driving video, the facial coefficient features and the expression generation model comprises: Performing convolution coding on the target training driving video and the facial coefficient feature respectively to obtain a second coding feature and a third coding feature; The second coding feature and the third coding feature are subjected to feature fusion by a second process to obtain a second fused feature; the second process comprises an adaptive normalization process with an attention mechanism; Performing residual calculation on the second fused feature to obtain a first residual feature; Performing deconvolution processing on the first residual feature to obtain a first deconvolution feature; Performing a third processing on the first deconvolution feature to obtain a training driven image; the third processing includes extracting deep features from the first deconvolution feature by using a spatial normalization process, and uniformly and adaptively adjusting a normalization parameter of the spatial normalization process; Performing video encoding according to the training driven image to obtain a training target video; The training target video is converged by using an L1 loss function until the expression generation model meets the second requirement, thereby obtaining the target expression generation model.

10. A method for generating expression of a digital human according to claim 9, characterized in that: The second process comprises: The second coding feature and the third coding feature are divided into a plurality of subsequences, feature coding processing is performed on each subsequence, attention weights corresponding to different subsequences are calculated by a spatial attention module, and a plurality of feature coding processing results are fused based on the attention weight calculation results to obtain local feature coding; the spatial attention module is preset in the expression generation model; The local feature code is divided into multiple subsequences, feature coding is performed on each subsequence, attention weights corresponding to different subsequences are calculated by the spatial attention module, and multiple feature coding processing results are fused based on the attention weight calculation results to obtain the local feature code; Iteratively updating, and independently outputting the local feature codes obtained in each round to obtain multiple local feature codes; Attention weights are calculated for the multiple local feature codes according to the channel attention module, and feature processing is performed based on the attention weight calculation results to obtain a global feature code; the multiple local feature codes are fused with the global feature code to obtain the second fused feature; the channel attention module is preset in the expression generation model.

11. A method for generating digital human expression according to claim 9, characterized in that: The step of using the L1 loss function to converge the training generated video includes: Calculating a third error between the training target video and the target expression video using the L1 loss function; When the third error meets the second requirement, the training of the expression generation model is stopped to obtain the target expression generation model.

Images