Digital human expression driving method and device, storage medium and computer equipment

Abstract

According to the digital human expression driving method and device, the storage medium and the computer equipment provided by the invention, when expression driving is carried out on a digital human, the target audio and the target text are firstly obtained, and the target emotion feature extraction model is adopted to carry out feature extraction on the target audio and the target text to obtain the multi-modal features and the corresponding emotion categories; the expression driving accuracy is improved through the multi-modal information; then, character expression data corresponding to the emotion category and a target expression parameter prediction model are determined, and a cross-space-time attention mechanism is introduced into the model, so that an expression parameter sequence generated after expression parameter prediction is carried out on the multi-modal features, the emotion category and the character expression data by the model comprises feature association operation of a space-time level; therefore, after the target person image is acquired, the target human face generation model can be adopted to perform expression driving on the target person image based on the expression parameter sequence, and a digital human expression driving video with time and space continuity is generated.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a digital human expression driving method, device, storage medium and computer equipment. Background Art

[0002] With the rapid advancement of digital human technology, its application scope is also expanding, and it brings people and machines closer in an unprecedented interactive form. In daily communication, the human face is an important social medium, and the accurate communication of its expressions and emotions is crucial to digital human technology. The communication process of expressions and emotions mainly relies on motion capture equipment or facial reconstruction technology to obtain expression parameters, and then realize the dynamic changes of expressions through driving algorithms.

[0003] At present, digital human expression driving methods mostly rely on deep neural networks, extracting expression features from text or audio data, and then shaping expressions through image generation technology. However, these methods are limited to feature processing at the image level and lack in-depth correlation operations in the spatiotemporal dimension, so they cannot achieve high-quality video-level digital human expressions. Summary of the invention

[0004] The purpose of this application is to solve at least one of the above-mentioned technical defects, especially the technical defect that the digital human expression driving method in the prior art is limited to feature processing at the image level and lacks in-depth correlation operations in the time and space dimensions, and therefore cannot achieve high-quality video-level digital human expressions.

[0005] The present application provides a digital human expression driving method, the method comprising:

[0006] Acquire a target audio and a target text corresponding to the target audio, and use a target emotion feature extraction model to perform feature extraction on the target audio and the target text to obtain a multimodal feature and an emotion category corresponding to the multimodal feature;

[0007] Determine the character expression data corresponding to the emotion category, and determine a target expression parameter prediction model; the target expression parameter prediction model is trained using a cross-temporal and spatial attention mechanism;

[0008] Inputting the multimodal features, the emotion categories and the character expression data into the target expression parameter prediction model to obtain an expression parameter sequence output by the target expression parameter prediction model;

[0009] A target person image is obtained, and a target person face generation model is used to perform expression driving on the target person image based on the expression parameter sequence to generate a digital human expression driven video.

[0010] Optionally, the target emotion feature extraction model includes an audio feature extraction layer, a text feature extraction layer, a multimodal feature fusion layer and a classification layer;

[0011] The target emotion feature extraction model is used to extract features from the target audio and the target text to obtain multimodal features and emotion categories corresponding to the multimodal features, including:

[0012] Performing feature extraction on the target audio through the audio feature extraction layer to obtain audio features, and performing feature extraction on the target text through the text feature extraction layer to obtain text features;

[0013] Using the multimodal feature fusion layer to concatenate and fuse the audio features and the text features, and output to obtain a multimodal feature;

[0014] The multimodal features are input into the classification layer so that the classification layer performs emotion classification on the multimodal features and outputs the emotion category.

[0015] Optionally, determining the character expression data corresponding to the emotion category includes:

[0016] Retrieving the emotion category in an expression database to obtain character expression parameters corresponding to the emotion category;

[0017] Wherein, an index structure of multiple character expression parameters is pre-established in the expression database.

[0018] Optionally, the process of constructing the expression database includes:

[0019] Collecting a character data set from multiple data sources, and marking the emotion category corresponding to each character data in the character data set;

[0020] For each emotion category, face reconstruction technology is used to perform three-dimensional recognition on multiple face data under the emotion category to obtain the face key points of each face data, and the character expression parameters corresponding to the emotion category are determined based on each face key point;

[0021] An index structure is established between each emotion category and the corresponding character expression parameter, and an expression database is constructed based on each index structure.

[0022] Optionally, determining the target expression parameter prediction model includes:

[0023] Inputting the pre-acquired sample character emotion data into a preset initial expression parameter prediction model to obtain a predicted expression parameter sequence output by the initial expression parameter prediction model; wherein the sample character emotion data includes multimodal features, emotion categories and character expression data;

[0024] The predicted expression parameter sequence is targeted to be close to the real expression parameter sequence corresponding to the sample character emotion data, and the initial expression parameter prediction model is trained using a cross-temporal and spatial attention mechanism;

[0025] When the initial expression parameter prediction model meets the preset training end condition, the trained initial expression parameter prediction model is used as the target expression parameter prediction model.

[0026] Optionally, the target expression parameter prediction model includes a text branch network, a video branch network, a CrossAttention network and a fully connected layer;

[0027] The step of inputting the multimodal features, the emotion category and the character expression data into the target expression parameter prediction model to obtain an expression parameter sequence output by the target expression parameter prediction model comprises:

[0028] Performing high-dimensional mapping on the multimodal features and the emotion categories through the text branch network to obtain text branch features, and performing high-dimensional mapping on the character expression data through the video branch network to obtain video branch features;

[0029] The Cross Attention network is used to calculate the cross-modal attention weights of the text branch features, the video branch features, the multimodal features, and the character expression data, and outputs the cross-modal attention features;

[0030] The cross-modal attention features and the character expression data are input into the fully connected layer to obtain an expression parameter sequence output by the fully connected layer.

[0031] Optionally, the step of using the target human face generation model to drive the target human image by expression based on the expression parameter sequence to generate a digital human expression driven video includes:

[0032] The expression parameter sequence and the target person image are input into a target face generation model, so that the target face generation model extracts parameter sequence features of the expression parameter sequence and image features of the target person image, and uses a cross-temporal and spatial attention mechanism to adjust the weights of the parameter sequence features and the image features, and, based on the adjustment results, the target person image is expression-driven to generate a digital human expression-driven video.

[0033] The present application also provides a digital human expression driving device, comprising:

[0034] A feature extraction module is used to obtain a target audio and a target text corresponding to the target audio, and use a target emotion feature extraction model to extract features from the target audio and the target text to obtain multimodal features and emotion categories corresponding to the multimodal features;

[0035] A model determination module, used to determine the character expression data corresponding to the emotion category, and to determine a target expression parameter prediction model; the target expression parameter prediction model is obtained by training with a cross-temporal and spatial attention mechanism;

[0036] A parameter prediction module, used for inputting the multimodal features, the emotion category and the character expression data into the target expression parameter prediction model to obtain an expression parameter sequence output by the target expression parameter prediction model;

[0037] The expression driving module is used to obtain a target person image, and use a target face generation model to perform expression driving on the target person image based on the expression parameter sequence to generate a digital human expression driving video.

[0038] The present application also provides a storage medium, in which computer-readable instructions are stored. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the digital human expression driving method as described in any of the above embodiments.

[0039] The present application also provides a computer device, comprising: one or more processors, and a memory;

[0040] The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the digital human expression driving method described in any one of the above embodiments are executed.

[0041] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0042] The digital human expression driving method, device, storage medium and computer equipment provided by the present application can first obtain the target audio and the target text corresponding to the target audio when driving the digital human expression, and use the target emotion feature extraction model to extract the target audio and target text, so as to obtain the multimodal features and the emotion categories corresponding to the multimodal features, so as to realize the emotion feature extraction through multimodal information, thereby improving the accuracy of the digital human expression driving; then the human expression data corresponding to the emotion category and the target expression parameter prediction model can be determined. Since the target expression parameter prediction model introduces a cross-temporal and spatial attention mechanism, the expression parameter sequence generated by the model after the expression parameter prediction of the multimodal features, emotion categories and human expression data contains the feature association operation at the spatiotemporal level. Therefore, after obtaining the target human image, the present application can use the target human face generation model to drive the target human image based on the expression parameter sequence to generate a digital human expression driving video with temporal and spatial continuity. In summary, the present application can realize the expression parameter prediction capability with video level by combining multimodal input information and cross-temporal and spatial attention mechanism, thereby improving the continuity of digital human expression driving in time and space in video scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0044] Figure 1 A schematic diagram of a flow chart of a digital human expression driving method provided in an embodiment of the present application;

[0045] Figure 2 A schematic diagram of a process of applying a target emotion feature extraction model provided in an embodiment of the present application;

[0046] Figure 3 A schematic diagram of a process of applying a target expression parameter prediction model provided in an embodiment of the present application;

[0047] Figure 4 A schematic diagram of the structure of a digital human expression driving device provided in an embodiment of the present application;

[0048] Figure 5 A schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0050] At present, digital human expression driving methods mostly rely on deep neural networks, extracting expression features from text or audio data, and then shaping expressions through image generation technology. However, these methods are limited to feature processing at the image level and lack in-depth correlation operations in the spatiotemporal dimension, so they cannot achieve high-quality video-level digital human expressions.

[0051] Based on this, this application proposes the following technical solutions, see below for details:

[0052] In one embodiment, Figure 1 As shown, Figure 1 A flowchart of a digital human expression driving method provided in an embodiment of the present application; the present application provides a digital human expression driving method, which specifically includes the following:

[0053] S110: Obtain target audio and target text corresponding to the target audio, and use a target emotion feature extraction model to extract features from the target audio and target text to obtain multimodal features and emotion categories corresponding to the multimodal features.

[0054] In this step, when it is necessary to configure the corresponding digital human expression drive for film-level audio, the computer device can first obtain the target audio and the target text corresponding to the target audio, and then use the pre-trained target emotion feature model to extract features from the target audio and target text, and obtain multimodal features and the emotion categories corresponding to the multimodal features, so that the accuracy of the digital human expression drive can be improved by using multimodal information.

[0055] Among them, the target emotion feature model refers to a model that uses machine learning or deep neural network learning as a pre-training model and is improved and trained. It is mainly used to analyze and extract the emotion features of relevant data such as audio and text, and then accurately infer the emotion categories in the audio and text based on the extracted emotion features, so as to drive the expression of the digital person according to this emotion information, making the driving effect more vivid and realistic.

[0056] Specifically, the target emotion feature model of the present application can extract multimodal features representing emotions from the target audio and target text, such as the pitch, volume, rhythm and other features in the target audio, as well as the emotional vocabulary, sentence structure and other features in the text, all of which can provide fine-grained information related to emotions. In addition, the model can also map the extracted multimodal features to specific emotion categories, such as happiness, sadness, anger, etc., so that the digital human can show expressions that are highly matched with the emotions in expression drive. Therefore, through the joint analysis of multimodal features, the present application can capture more comprehensive emotional information, making expression drive more accurate and natural; for example, the tone in the target audio may express an emotion, and the text content can provide additional context to help the digital human show more delicate expressions.

[0057] Furthermore, the computer device can store the target emotion feature model after training it, so that when performing feature extraction later, the pre-stored target emotion feature model can be directly called to extract features of the target audio and target emotion. In addition, the target emotion feature model of the present application can select the ImageBind model for improvement and training, so as to use the improved ImageBind model to operate on deep features, realize the extraction of multimodal features and the recognition of emotion categories. The ImageBind model here refers to a multimodal deep learning model, which can map data of different modes to a shared embedding space, so that data can find natural similarities between different modes, and can also automatically align features of different modes, so that the model can perform feature alignment while extracting multimodal features, reduce information loss, and improve the quality of multimodal information integration.

[0058] S120: Determine the character expression data corresponding to the emotion category, and determine the target expression parameter prediction model; the target expression parameter prediction model is trained using a cross-temporal and spatial attention mechanism.

[0059] In this step, after obtaining the emotion category corresponding to the multimodality through step S110, the computer device can determine the character expression data corresponding to the emotion category, and determine the target expression parameter prediction model for predicting the expression parameters of the character expression data; because the target expression parameter prediction model is trained using a cross-temporal and spatial attention mechanism, the result predicted by the model includes feature association operations at the temporal and spatial levels, thereby ensuring the continuity of subsequent digital human expression driving in time and space.

[0060] Among them, human expression data refers to the expression information composed of multiple three-dimensional facial key points. These key points are the core data used to drive the digital human's expression, usually including the position and posture of the eyes, mouth, eyebrows and other parts, which can accurately describe the slight changes in facial features, thereby reflecting a specific emotional state. The target expression parameter prediction model is a model trained using a cross-temporal and spatial attention mechanism, which is used to predict and generate the expression parameters of the digital human in a time series, ensuring the continuity of the digital human's facial expression in time and space.

[0061] It is understandable that the cross-temporal and spatial attention mechanism of the present application is a mechanism for processing the spatiotemporal relationship in data, which extends the traditional attention mechanism to the time and space dimensions. Therefore, based on the cross-temporal and spatial attention mechanism, the target expression parameter prediction model can capture the correlation between feature data in the two dimensions of time and space, so as to understand the dynamic changes of the object and the mutual influence of spatial features, and ensure the coherence and naturalness of the model output results.

[0062] For example, in the time dimension, the target expression parameter prediction model can focus on the changes in data at different times to capture dynamic information; for example, when analyzing expressions, cross-temporal attention can focus on how a smile gradually unfolds from nothing to something, so that the model can understand the progress of events, the connection of actions, and the changes in expressions over time through the association of time. In the spatial dimension, the target expression parameter prediction model can focus on the spatial position and distribution relationship of the data to capture the interaction of objects at different spatial points at the same time; for example, when analyzing a specific expression composed of the coordination relationship between a person's eyebrows, eyes, and mouth, cross-temporal attention can assist the model in capturing the spatial interaction of features at the same time to understand the spatial distribution characteristics of expressions or actions.

[0063] S130: Inputting multimodal features, emotion categories and character expression data into a target expression parameter prediction model to obtain an expression parameter sequence output by the target expression parameter prediction model.

[0064] In this step, after obtaining the target expression parameter prediction model through step S120, the computer device can input multimodal features, emotion categories and character expression data into the target expression parameter prediction model to obtain the expression parameter sequence output by the target expression parameter prediction model. Since the target expression parameter prediction model introduces a cross-temporal and spatial attention mechanism, the expression parameter sequence output by the model is coherent and natural in time and space.

[0065] It is understandable that the expression parameter sequence, as the output result of the target expression parameter prediction model, can drive the digital human's facial expression to present dynamic effects that conform to the emotion category. The cross-temporal and spatial attention mechanism introduced by the target expression parameter prediction model can focus on the changing trend of expressions between different time points in the time dimension, and focus on the relationship and linkage between key points of the three-dimensional face in the spatial dimension. Therefore, the emotional expression in the expression parameter sequence output by the model can achieve a high degree of coherence in time and space when matched with the multimodal features and the character's expression data, ensuring the smooth transition and natural expression of expression changes, so that the digital human expression drive presents a more realistic and vivid effect in continuous multi-frame images.

[0066] S140: Acquire a target person image, and use a target face generation model to perform expression driving on the target person image based on an expression parameter sequence to generate a digital human expression driven video.

[0067] In this step, after obtaining the expression parameter sequence through step S130, the computer device can obtain the target person image as a digital human for expression driving, and then use the target face generation model to drive the target person image based on the expression parameter sequence to generate a digital human expression driven video with temporal and spatial continuity.

[0068] It is understandable that the target face generation model of the present application can be improved and trained using the U-Net model as a pre-training model. The U-Net model is a deep learning model for generating segmented outputs that can make accurate pixel-level predictions on details, while using skip connections to pass information from early layers in the encoder directly to the decoder, thereby retaining more spatial details.

[0069] Furthermore, the present application can further improve and train the structure of the U-Net model to better adapt to the spatiotemporal characteristics of expression changes in the expression parameter sequence. Driven by the expression parameter sequence provided by the target expression parameter prediction model, the U-Net model can accurately update the expression of the target person image, so that the digital human can show reasonable expression evolution in the time dimension, and the relationship between facial features such as eyes, eyebrows, mouth and other parts can be accurately mapped in space, ensuring that the final generated digital human expression-driven video has realism and fluency in facial expression.

[0070] In the above embodiment, when the digital human is driven by expression, the target audio and the target text corresponding to the target audio can be obtained first, and the target emotion feature extraction model can be used to extract the target audio and the target text to obtain multimodal features and the emotion categories corresponding to the multimodal features, so that the emotion feature extraction can be realized through multimodal information, thereby improving the accuracy of the digital human expression driving; then the human expression data corresponding to the emotion category and the target expression parameter prediction model can be determined. Since the target expression parameter prediction model introduces a cross-temporal and spatial attention mechanism, the expression parameter sequence generated by the model after the expression parameter prediction of the multimodal features, emotion categories and human expression data contains the feature association operation at the spatiotemporal level. Therefore, after obtaining the target human image, the present application can use the target human face generation model to drive the target human image based on the expression parameter sequence to generate a digital human expression driven video with temporal and spatial continuity. In summary, the present application can achieve the ability to predict expression parameters at the video level by combining multimodal input information and the cross-temporal and spatial attention mechanism, thereby improving the continuity of the digital human expression driving in time and space in the video scene.

[0071] In one embodiment, Figure 2 As shown, Figure 2 A schematic diagram of a process of applying a target emotion feature extraction model provided in an embodiment of the present application; Figure 2 In step S110, the target emotion feature extraction model may include an audio feature extraction layer, a text feature extraction layer, a multimodal feature fusion layer, and a classification layer; wherein the process of extracting features from the target audio and the target text using the target emotion feature extraction model to obtain multimodal features and emotion categories corresponding to the multimodal features may include:

[0072] S111: extracting features from the target audio through the audio feature extraction layer to obtain audio features, and extracting features from the target text through the text feature extraction layer to obtain text features.

[0073] S112: Use the multimodal feature fusion layer to concatenate and fuse the audio features and text features, and output the multimodal features.

[0074] S113: Input the multimodal features into the classification layer, so that the classification layer performs emotion classification on the multimodal features and outputs the emotion category.

[0075] In this embodiment, the target emotion extraction model can be composed of an audio feature extraction layer, a text feature extraction layer, a multimodal feature fusion layer and a classification layer. Therefore, when the target emotion feature extraction model is used to extract multimodal features, the computer device can first input the target audio into the audio feature extraction layer, and input the target text into the text feature extraction layer, so as to obtain the audio features output by the audio feature extraction layer and the text features output by the text feature extraction layer, and then the computer device can use the multimodal feature fusion layer to splice and fuse the audio features and the text features to obtain multimodal features, and then the multimodal features can be input into the classification layer, so that the classification layer can classify the multimodal features into emotions and output the obtained emotion categories.

[0076] It should be noted that the target emotion feature extraction model of the present application can be obtained by structurally improving the ImageBind model and then training it. Specifically, the computer device can set the feature extraction capabilities of the two modalities of audio and text in the multi-feature extraction layer of the ImageBind model, and then build a multimodal feature fusion layer based on the Transformer structure, which is used to perform feature splicing and feature extraction on the extracted audio and text features to obtain multimodal features with semantic information, whose feature dimension information can be 1x768, and the numerical type is a floating point number; finally, the computer device can create a new classification layer according to the number of emotion categories, and then perform emotion recognition classification based on the above-mentioned extracted multimodal features, so as to calculate the loss value according to the classification result, and perform back propagation based on the calculated loss value to update the model weights until the model meets the training end conditions.

[0077] In one embodiment, determining the character expression data corresponding to the emotion category in step S120 may include:

[0078] S121: Searching the emotion category in the expression database to obtain the character expression parameters corresponding to the emotion category.

[0079] In this embodiment, the computer device is configured with an expression database, in which a plurality of index structures of character expression parameters are pre-established. Therefore, the computer device can directly search for the emotion category in the expression database, and thus obtain the character expression parameters in the index structure corresponding to the emotion category.

[0080] It is understandable that the index structure in the expression database contains the corresponding human expression parameters under different emotion categories. These human expression parameters are accurately labeled and can accurately reflect the expression characteristics under different emotional states. When the system needs to generate an expression of a specific emotion category for a digital human, the computer device can directly perform a quick search in the expression database and extract the corresponding human expression parameters from the index structure based on the emotion category. Through this retrieval mechanism, the computer device can quickly obtain expression data that highly matches the emotion category and apply it to the expression drive of the digital human, thereby improving the efficiency of expression generation and reducing the calculation time of generating expression parameters from scratch.

[0081] In one embodiment, the process of constructing the expression database in step S121 may include:

[0082] S1211: Collect a character data set from multiple data sources, and mark the emotion category corresponding to each character data in the character data set.

[0083] S1212: For each emotion category, a face reconstruction technique is used to perform three-dimensional recognition on multiple face data under the emotion category to obtain face key points of each face data, and the character expression parameters corresponding to the emotion category are determined based on each face key point.

[0084] S1213: Establishing an index structure between each emotion category and the corresponding character expression parameter, and constructing an expression database according to each index structure.

[0085] In this embodiment, when constructing an expression database, the computer device may first collect multiple character data from multiple data sources to form a character data set, and then label each character data in the character data set with a corresponding emotion category. For each emotion category, the computer device may use face reconstruction technology to perform three-dimensional recognition on multiple face data under the emotion category, obtain the face key points of each face data, and determine the character expression parameters corresponding to the emotion category based on each face key point; finally, the computer device may establish an index structure between each emotion category and the corresponding character expression parameter, and construct an expression database based on each index structure.

[0086] Among them, face reconstruction technology refers to the technology of extracting three-dimensional face shape and structure information from two-dimensional images or videos through computer vision and image processing methods, and restoring and modeling facial details. It converts ordinary two-dimensional facial images into accurate three-dimensional face models and provides precise positioning of facial features such as eyes, nose, mouth and other parts.

[0087] It is understandable that computer equipment can not only provide the expression features of a person in a single image in the emotion category, but also integrate the expression features of different individuals to extract representative emotion features. After calculation and analysis, these features can form a set of high-dimensional character expression parameters. The character expression parameters cover the change amplitude, movement trajectory and mutual coordination of each key point on the face, providing a data basis for subsequent expression driving.

[0088] Furthermore, when the computer device suggests an index structure, it can be constructed through the ElasticSearch retrieval tool. Elastic Search uses Lucene's inverted index technology to achieve faster filtering than relational databases. Therefore, when using the Elastic Search retrieval tool, the character expression parameters corresponding to each emotion category can be obtained first, and then the Elastic Search retrieval tool can be used to establish an index between each emotion category and its character expression parameters, thereby obtaining an index structure in the expression database.

[0089] In one embodiment, the process of determining the target expression parameter prediction model in step S120 may include:

[0090] S122: Inputting the pre-acquired sample character emotion data into a preset initial expression parameter prediction model to obtain a predicted expression parameter sequence output by the initial expression parameter prediction model; wherein the sample character emotion data includes multimodal features, emotion categories and character expression data.

[0091] S123: The goal is to make the predicted expression parameter sequence close to the real expression parameter sequence corresponding to the sample character emotion data, and use the cross-temporal and spatial attention mechanism to train the initial expression parameter prediction model.

[0092] S124: When the initial expression parameter prediction model meets the preset training end condition, the trained initial expression parameter prediction model is used as the target expression parameter prediction model.

[0093] In this embodiment, the computer device can first obtain sample character emotion data composed of multimodal features, emotion categories and character expression data, as well as the real expression parameter sequence corresponding to the sample character emotion data, as basic data for model training. Then, the computer device can input the sample character emotion data into a preset initial expression parameter prediction model to obtain a predicted expression parameter sequence output by the initial expression parameter prediction model, and train the initial expression parameter prediction model using a cross-temporal and spatial attention mechanism with the goal of making the predicted expression parameter sequence approach the real expression parameter sequence corresponding to the sample character emotion data, until the initial expression parameter prediction model meets the preset training end condition, and the trained initial expression parameter prediction model is used as the target expression parameter prediction model.

[0094] Specifically, when the target expression parameter prediction model is training the model, the computer device can use different types of sample character emotion data as training samples, and label each training sample with a sample label, that is, the corresponding real expression parameter sequence. When all training samples are labeled, the computer device can input the training samples with sample labels into the preset initial expression parameter prediction model for forward propagation to train the model, and use the preset target loss function to tune the model parameters during the back propagation process of the model. When the model meets certain training conditions or parameter convergence conditions, such as the number of iterations reaching the set value, the training is considered to be completed, and the trained model can be used as the final target expression parameter prediction model.

[0095] Furthermore, after the computer device obtains the target expression parameter prediction model through training, it can store it so that when performing expression parameter prediction later, the pre-stored target expression parameter prediction model can be directly called to perform expression parameter prediction on multimodal features and character expression data. In addition, the target expression parameter prediction model of the present application can select the VindLU model for improvement and training; the VindLU model here is a multimodal model based on computer vision and natural language understanding, which can effectively process and integrate visual and language information, and improve the ability to understand multimodal data, thereby enhancing the model's recognition and response to emotional changes, so as to improve the overall effect of expression drive and ensure that the emotional expression of digital people is more delicate and real.

[0096] In one embodiment, Figure 3 As shown, Figure 3 A schematic diagram of a process of applying a target expression parameter prediction model provided in an embodiment of the present application; Figure 3In step S130, the target expression parameter prediction model may include a text branch network, a video branch network, a Cross Attention network, and a fully connected layer; wherein the multimodal features, emotion categories, and character expression data are input into the target expression parameter prediction model to obtain the expression parameter sequence output by the target expression parameter prediction model, which may include:

[0097] S131: Perform high-dimensional mapping of multimodal features and emotion categories through a text branch network to obtain text branch features, and perform high-dimensional mapping of character expression data through a video branch network to obtain video branch features.

[0098] S132: The Cross Attention network is used to calculate the cross-modal attention weights of text branch features, video branch features, multimodal features, and character expression data, and the cross-modal attention features are output.

[0099] S133: Input the cross-modal attention features and character expression data into the fully connected layer to obtain the expression parameter sequence output by the fully connected layer.

[0100] In this embodiment, the target expression parameter prediction model can be composed of a text branch network, a video branch network, a CrossAttention network and a fully connected layer. Therefore, when predicting the expression parameter sequence, the computer device can first input the multimodal features and emotion categories into the text branch network, and input the character expression data into the video branch network, so as to obtain the text branch features output by the text branch network and the video branch features output by the video branch network; then the computer device can use the Cross Attention network to calculate the cross-modal attention weights of the text branch features, video branch features, multimodal features and character expression data to output the cross-modal attention features, and input the cross-modal attention features and character expression data into the fully connected layer to obtain the expression parameter sequence output by the fully connected layer.

[0101] It should be noted that the target expression parameter prediction model of the present application can be obtained by structurally improving the VindLU model and then training it. Specifically, the computer device can set the first-layer structural input of the text branch network of the VindLU model to the token value corresponding to the multimodal features and emotion labels of 1x768 dimensions, and set the first-layer structural input of the video branch network to the character expression data; the character expression data here is the coordinate value of the key points of the three-dimensional face, and its dimension information is N×3, where N represents the number of key points, and 3 represents the coordinate values ​​in the three directions of x, y, and z; then, the computer device can also set the Cross Attention network in the structurally modified VindLU model, so that it can combine multimodal features and character expression data to calculate the cross-modal attention weight, and use the calculated weight in subsequent feature operations, so that the model has the ability to extract attention features at the spatiotemporal level; finally, the computer device can create a new fully connected layer in the model, and its output dimension is set to B×N×3, which is used to represent the expression parameter sequence predicted and output by the model, where B represents the number of frames.

[0102] In one embodiment, the process of using the target face generation model to drive the target person image based on the expression parameter sequence in step S140 to generate a digital human expression driven video may include:

[0103] S141: Inputting the expression parameter sequence and the target person image into the target face generation model, so that the target face generation model extracts the parameter sequence features of the expression parameter sequence and the image features of the target person image, and uses a cross-temporal and spatial attention mechanism to adjust the weights of the parameter sequence features and the image features, and, based on the adjustment results, drives the target person image for expression, and generates a digital human expression-driven video.

[0104] In this embodiment, when the computer device performs expression driving on the data person, it can first obtain the target face generation model, so that the expression parameter sequence and the target person image can be input into the target face generation model, so that the target face generation model can extract the parameter sequence features of the expression parameter sequence and the image features of the target person image, and use the cross-temporal and spatial attention mechanism to adjust the weights of the parameter sequence features and the image features. Finally, the target face generation model can perform expression driving on the target person image according to the adjustment results, and generate and output the digital human expression driven video.

[0105] It should be noted that the target expression parameter prediction model of the present application can be obtained by structurally improving the U-Net model and then training it. Specifically, the computer device can modify the shallow structure of the U-Net model, and set its input to the target person image and expression parameter sequence, wherein the dimension of the target person image can be Bx3xHxW, and the dimension of the expression parameter sequence can be BxNx3, wherein B represents the number of image frames, N represents the number of three-dimensional face key points, H represents the image height, and W represents the image width; then, the computer device can also introduce the CrossAttention module into the structurally modified U-Net model, so that the model can calculate the cross-modal attention weights by combining the image features and the parameter sequence features, and use the calculated weights for subsequent feature operations; finally, the computer device can set the output dimension of the output layer in the network structure of the U-Net model to Bx3xHxW, which is used to represent the digital human expression-driven video generated by the model consisting of multiple frames of character images.

[0106] The digital human expression driving device provided in the embodiment of the present application is described below. The digital human expression driving device described below and the digital human expression driving method described above can be referenced to each other.

[0107] In one embodiment, Figure 4 As shown, Figure 4 A schematic diagram of the structure of a digital human expression driving device provided in an embodiment of the present application; the present application also provides a digital human expression driving device, including a feature extraction module 210, a model determination module 220, a parameter prediction module 230 and an expression driving module 240, specifically including the following:

[0108] The feature extraction module 210 is used to obtain the target audio and the target text corresponding to the target audio, and use the target emotion feature extraction model to extract features from the target audio and the target text to obtain multimodal features and emotion categories corresponding to the multimodal features.

[0109] The model determination module 220 is used to determine the character expression data corresponding to the emotion category and determine the target expression parameter prediction model; the target expression parameter prediction model is trained using a cross-temporal and spatial attention mechanism.

[0110] The parameter prediction module 230 is used to input the multimodal features, emotion categories and character expression data into the target expression parameter prediction model to obtain the expression parameter sequence output by the target expression parameter prediction model.

[0111] The expression driving module 240 is used to obtain a target person image, and use a target face generation model to perform expression driving on the target person image based on an expression parameter sequence to generate a digital human expression driven video.

[0112] In the above embodiment, when the digital human is driven by expression, the target audio and the target text corresponding to the target audio can be obtained first, and the target emotion feature extraction model can be used to extract the target audio and the target text to obtain multimodal features and the emotion categories corresponding to the multimodal features, so that the emotion feature extraction can be realized through multimodal information, thereby improving the accuracy of the digital human expression driving; then the human expression data corresponding to the emotion category and the target expression parameter prediction model can be determined. Since the target expression parameter prediction model introduces a cross-temporal and spatial attention mechanism, the expression parameter sequence generated by the model after the expression parameter prediction of the multimodal features, emotion categories and human expression data contains the feature association operation at the spatiotemporal level. Therefore, after obtaining the target human image, the present application can use the target human face generation model to drive the target human image based on the expression parameter sequence to generate a digital human expression driven video with temporal and spatial continuity. In summary, the present application can achieve the ability to predict expression parameters at the video level by combining multimodal input information and the cross-temporal and spatial attention mechanism, thereby improving the continuity of the digital human expression driving in time and space in the video scene.

[0113] In one embodiment, the target emotion feature extraction model in the feature extraction module 210 may include an audio feature extraction layer, a text feature extraction layer, a multimodal feature fusion layer, and a classification layer; wherein the feature extraction module 210 may also include:

[0114] The audio-text feature extraction submodule is used to extract features from the target audio through the audio feature extraction layer to obtain audio features, and to extract features from the target text through the text feature extraction layer to obtain text features.

[0115] The feature concatenation and fusion submodule is used to concatenate and fuse audio features and text features using the multimodal feature fusion layer, and output multimodal features.

[0116] The emotion classification submodule is used to input the multimodal features into the classification layer so that the classification layer can classify the multimodal features into emotions and output the emotion categories.

[0117] In one embodiment, the model determination module 220 may include:

[0118] The category retrieval submodule is used to retrieve the emotion category in the expression database and obtain the character expression parameters corresponding to the emotion category.

[0119] Wherein, an index structure of multiple character expression parameters is pre-established in the expression database.

[0120] In one embodiment, the category search submodule may include:

[0121] The data collection unit is used to collect character data sets from multiple data sources and mark the emotion category corresponding to each character data in the character data sets.

[0122] The parameter calculation unit is used to perform three-dimensional recognition on multiple face data under each emotion category using face reconstruction technology, obtain the face key points of each face data, and determine the character expression parameters corresponding to the emotion category based on each face key point.

[0123] The index building unit is used to build an index structure between each emotion category and the corresponding character expression parameter, and to build an expression database according to each index structure.

[0124] In one embodiment, the model determination module 220 may further include:

[0125] The model prediction submodule is used to input the pre-acquired sample character emotion data into the preset initial expression parameter prediction model to obtain the predicted expression parameter sequence output by the initial expression parameter prediction model; wherein the sample character emotion data includes multimodal features, emotion categories and character expression data.

[0126] The model training submodule is used to train the initial expression parameter prediction model with the goal of making the predicted expression parameter sequence close to the real expression parameter sequence corresponding to the sample character's emotional data, and adopting the cross-temporal and spatial attention mechanism.

[0127] The model generation submodule is used to use the trained initial expression parameter prediction model as the target expression parameter prediction model when the initial expression parameter prediction model meets the preset training end condition.

[0128] In one embodiment, the target expression parameter prediction model in the parameter prediction module 230 may include a text branch network, a video branch network, a Cross Attention network, and a fully connected layer; wherein the parameter prediction module 230 may also include:

[0129] The high-dimensional mapping submodule is used to perform high-dimensional mapping of multimodal features and emotion categories through a text branch network to obtain text branch features, and to perform high-dimensional mapping of character expression data through a video branch network to obtain video branch features.

[0130] The weight adjustment submodule is used to calculate the cross-modal attention weights of text branch features, video branch features, multimodal features and character expression data using the Cross Attention network, and output the cross-modal attention features.

[0131] The sequence output submodule is used to input the cross-modal attention features and character expression data into the fully connected layer to obtain the expression parameter sequence output by the fully connected layer.

[0132] In one embodiment, the expression driving module 240 may include:

[0133] The model-driven submodule is used to input the expression parameter sequence and the target person image into the target face generation model, so that the target face generation model can extract the parameter sequence features of the expression parameter sequence and the image features of the target person image, and use the cross-temporal and spatial attention mechanism to adjust the weights of the parameter sequence features and the image features, and, according to the adjustment results, perform expression driving on the target person image to generate a digital human expression-driven video.

[0134] In one embodiment, the present application further provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the digital human expression driving method as described in any of the above embodiments.

[0135] In one embodiment, the present application further provides a computer device having computer-readable instructions stored therein. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the digital human expression driving method as described in any one of the above embodiments.

[0136] Indicatively, Figure 5 As shown, Figure 5 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. The computer device 300 may be provided as a server. Figure 5 The computer device 300 includes a processing component 302, which further includes one or more processors, and a memory resource represented by a memory 301, for storing instructions executable by the processing component 302, such as an application. The application stored in the memory 301 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 302 is configured to execute instructions to perform the digital human expression driving method of any of the above embodiments.

[0137] The computer device 300 may further include a power supply component 303 configured to perform power management of the computer device 300, a wired or wireless network interface 304 configured to connect the computer device 300 to a network, and an input / output (I / O) interface 305. The computer device 300 may operate based on an operating system stored in the memory 301, such as Windows Server TM, Mac OS X TM, Unix TM, Linux TM, Free BSD TM, or the like.

[0138] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0139] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0140] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

[0141] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A digital human expression driving method, characterized in that: The method comprises: Acquire a target audio and a target text corresponding to the target audio, and use a target emotion feature extraction model to perform feature extraction on the target audio and the target text to obtain multimodal features and emotion categories corresponding to the multimodal features; Determine the character expression data corresponding to the emotion category, and determine a target expression parameter prediction model; the target expression parameter prediction model is trained using a cross-temporal and spatial attention mechanism; Inputting the multimodal features, the emotion categories and the character expression data into the target expression parameter prediction model to obtain an expression parameter sequence output by the target expression parameter prediction model; A target person image is obtained, and a target person face generation model is used to perform expression driving on the target person image based on the expression parameter sequence to generate a digital human expression driven video.

2. The digital human expression driving method according to claim 1, characterized in that: The target emotion feature extraction model includes an audio feature extraction layer, a text feature extraction layer, a multimodal feature fusion layer and a classification layer; The target emotion feature extraction model is used to extract features from the target audio and the target text to obtain multimodal features and emotion categories corresponding to the multimodal features, including: Performing feature extraction on the target audio through the audio feature extraction layer to obtain audio features, and performing feature extraction on the target text through the text feature extraction layer to obtain text features; Using the multimodal feature fusion layer to concatenate and fuse the audio features and the text features, and output to obtain a multimodal feature; The multimodal features are input into the classification layer so that the classification layer performs emotion classification on the multimodal features and outputs the emotion category.

3. The digital human expression driving method according to claim 1, characterized in that: The determining of the character expression data corresponding to the emotion category includes: Retrieving the emotion category in an expression database to obtain character expression parameters corresponding to the emotion category; Wherein, an index structure of multiple character expression parameters is pre-established in the expression database.

4. The digital human expression driving method according to claim 3, characterized in that: The process of constructing the expression database includes: Collecting a character data set from multiple data sources, and marking the emotion category corresponding to each character data in the character data set; For each emotion category, face reconstruction technology is used to perform three-dimensional recognition on multiple face data under the emotion category to obtain the face key points of each face data, and the character expression parameters corresponding to the emotion category are determined based on each face key point; An index structure is established between each emotion category and the corresponding character expression parameter, and an expression database is constructed based on each index structure.

5. The digital human expression driving method according to claim 1, characterized in that: Determining the target expression parameter prediction model includes: Inputting the pre-acquired sample character emotion data into a preset initial expression parameter prediction model to obtain a predicted expression parameter sequence output by the initial expression parameter prediction model; wherein the sample character emotion data includes multimodal features, emotion categories and character expression data; The predicted expression parameter sequence is targeted to be close to the real expression parameter sequence corresponding to the sample character emotion data, and the initial expression parameter prediction model is trained using a cross-temporal and spatial attention mechanism; When the initial expression parameter prediction model meets the preset training end condition, the trained initial expression parameter prediction model is used as the target expression parameter prediction model.

6. The digital human expression driving method according to claim 1, characterized in that: The target expression parameter prediction model includes a text branch network, a video branch network, a Cross Attention network and a fully connected layer; The step of inputting the multimodal features, the emotion category and the character expression data into the target expression parameter prediction model to obtain an expression parameter sequence output by the target expression parameter prediction model comprises: Performing high-dimensional mapping on the multimodal features and the emotion categories through the text branch network to obtain text branch features, and performing high-dimensional mapping on the character expression data through the video branch network to obtain video branch features; The Cross Attention network is used to calculate the cross-modal attention weights of the text branch features, the video branch features, the multimodal features, and the character expression data, and outputs the cross-modal attention features; The cross-modal attention features and the character expression data are input into the fully connected layer to obtain an expression parameter sequence output by the fully connected layer.

7. The digital human expression driving method according to claim 1, characterized in that: The method of using the target human face generation model to drive the target human image by expression based on the expression parameter sequence to generate a digital human expression driven video includes: The expression parameter sequence and the target person image are input into a target face generation model, so that the target face generation model extracts parameter sequence features of the expression parameter sequence and image features of the target person image, and uses a cross-temporal and spatial attention mechanism to adjust the weights of the parameter sequence features and the image features, and, based on the adjustment results, the target person image is expression-driven to generate a digital human expression-driven video.

8. A digital human expression driving device, characterized in that: include: A feature extraction module is used to obtain a target audio and a target text corresponding to the target audio, and use a target emotion feature extraction model to perform feature extraction on the target audio and the target text to obtain a multimodal feature and an emotion category corresponding to the multimodal feature; A model determination module is used to determine the character expression data corresponding to the emotion category and determine a target expression parameter prediction model; the target expression parameter prediction model is trained using a cross-temporal and spatial attention mechanism; A parameter prediction module, used for inputting the multimodal features, the emotion category and the character expression data into the target expression parameter prediction model to obtain an expression parameter sequence output by the target expression parameter prediction model; The expression driving module is used to obtain a target person image, and use a target face generation model to perform expression driving on the target person image based on the expression parameter sequence to generate a digital human expression driving video.

9. A storage medium, characterized in that: The storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the digital human expression driving method as claimed in any one of claims 1 to 7.

10. A computer device, characterized in that: include: one or more processors, and memory; The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the digital human expression driving method according to any one of claims 1 to 7 are executed.

Images